2023-12-19T18:51:31.780 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T18:51:31.780 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329 --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329/orig.config.yaml 2023-12-19T18:51:31.835 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T18:51:31.919 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: null first_in_suite: false job_id: '7497329' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIshYVrt2DKHlTX/Sdu2dvCpDQ2/Iy9qOzMXB3cvi+FK4M2A6iNfCYWwz6OgsT3BhpLadM6UfQYwSI4nxbTwNw= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9YjkQCn2MGGsyTxyOuNT5q8xy0ZK7SdXyzUPvjnz69s0cohuF7xFzefH1fvVGlT8uDbMAex4e1Xs6nPfrfcZE= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_15:49:15 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-19T18:51:31.919 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-19T18:51:31.920 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-19T18:51:31.921 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T18:51:31.922 INFO:teuthology.task.internal:Checking packages... 2023-12-19T18:51:31.947 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-19T18:51:31.947 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T18:51:31.947 INFO:teuthology.packaging:ref: None 2023-12-19T18:51:31.947 INFO:teuthology.packaging:tag: None 2023-12-19T18:51:31.947 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:51:31.947 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T18:51:31.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-19T18:51:32.095 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202.gc7586e95 2023-12-19T18:51:32.097 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T18:51:32.103 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T18:51:32.103 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T18:51:32.109 INFO:teuthology.task.internal:Saving configuration 2023-12-19T18:51:32.129 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T18:51:32.150 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T18:51:32.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:46:55.890668', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIshYVrt2DKHlTX/Sdu2dvCpDQ2/Iy9qOzMXB3cvi+FK4M2A6iNfCYWwz6OgsT3BhpLadM6UfQYwSI4nxbTwNw='} 2023-12-19T18:51:32.194 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:46:55.892333', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9YjkQCn2MGGsyTxyOuNT5q8xy0ZK7SdXyzUPvjnz69s0cohuF7xFzefH1fvVGlT8uDbMAex4e1Xs6nPfrfcZE='} 2023-12-19T18:51:32.195 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-19T18:51:32.201 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-19T18:51:32.201 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-19T18:51:32.201 INFO:teuthology.run_tasks:Running task console_log... 2023-12-19T18:51:32.280 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f398b857f70>, signals=[15]) 2023-12-19T18:51:32.280 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-19T18:51:32.287 INFO:teuthology.task.internal:Opening connections... 2023-12-19T18:51:32.287 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T18:51:32.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:51:32.387 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2023-12-19T18:51:32.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:51:32.472 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-19T18:51:32.479 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2023-12-19T18:51:32.497 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2023-12-19T18:51:32.497 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2023-12-19T18:51:32.554 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="CentOS Stream" 2023-12-19T18:51:32.554 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="9" 2023-12-19T18:51:32.554 INFO:teuthology.orchestra.run.smithi055.stdout:ID="centos" 2023-12-19T18:51:32.554 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="rhel fedora" 2023-12-19T18:51:32.554 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="9" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el9" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:LOGO="fedora-logo-icon" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://centos.org/" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-19T18:51:32.555 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-19T18:51:32.556 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2023-12-19T18:51:32.574 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2023-12-19T18:51:32.591 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2023-12-19T18:51:32.591 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-19T18:51:32.647 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-19T18:51:32.648 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-19T18:51:32.648 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-12-19T18:51:32.671 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-19T18:51:32.686 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-19T18:51:32.692 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-19T18:51:32.692 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2023-12-19T18:51:32.696 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2023-12-19T18:51:32.711 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-19T18:51:32.717 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-19T18:51:32.717 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2023-12-19T18:51:32.753 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2023-12-19T18:51:32.779 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-19T18:51:32.832 INFO:teuthology.run_tasks:Running task kernel... 2023-12-19T18:51:32.854 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-19T18:51:32.854 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-19T18:51:32.854 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T18:51:32.855 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T18:51:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:51:32.869 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-12-19T18:51:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:51:32.871 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-12-19T18:51:32.924 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-361.el9.x86_64 2023-12-19T18:51:32.924 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-361.el9.x86_64 2023-12-19T18:51:32.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2023-12-19T18:51:32.928 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-361.el9.x86_64 2023-12-19T18:51:32.928 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.14.0-361.el9.x86_64 2023-12-19T18:51:32.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum install -y kernel 2023-12-19T18:51:35.036 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-19T18:51:35.146 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-19T18:51:36.580 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2023-12-19T18:51:36.687 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2023-12-19T18:51:38.310 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 6.1 MB/s | 7.9 MB 00:01 2023-12-19T18:51:39.071 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 7.9 MB 00:02 2023-12-19T18:51:45.930 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 18 MB 00:05 2023-12-19T18:51:50.834 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-19T18:51:52.173 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 15 kB 00:00 2023-12-19T18:51:54.068 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-19T18:51:55.493 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 18 MB 00:15 2023-12-19T18:51:58.866 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-12-19T18:52:00.792 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 5.8 MB 00:01 2023-12-19T18:52:01.123 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-19T18:52:01.123 INFO:teuthology.orchestra.run.smithi055.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-19T18:52:01.230 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T18:52:01.238 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout:Install 4 Packages 2023-12-19T18:52:01.239 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:52:01.240 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 96 M 2023-12-19T18:52:01.240 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 121 M 2023-12-19T18:52:01.241 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T18:52:02.227 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2023-12-19T18:52:03.284 INFO:teuthology.orchestra.run.smithi055.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 2.8 MB/s | 5.6 MB 00:01 2023-12-19T18:52:03.789 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2023-12-19T18:52:04.884 INFO:teuthology.orchestra.run.smithi055.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 5.7 MB/s | 20 MB 00:03 2023-12-19T18:52:07.192 INFO:teuthology.orchestra.run.smithi055.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 6.4 MB/s | 38 MB 00:05 2023-12-19T18:52:08.267 INFO:teuthology.orchestra.run.smithi055.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 6.5 MB/s | 32 MB 00:04 2023-12-19T18:52:08.267 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:52:08.268 INFO:teuthology.orchestra.run.smithi055.stdout:Total 14 MB/s | 96 MB 00:07 2023-12-19T18:52:08.528 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-19T18:52:09.158 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T18:52:09.573 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T18:52:09.574 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T18:52:10.768 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-19T18:52:10.768 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-19T18:52:10.859 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T18:52:10.864 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T18:52:10.864 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-12-19T18:52:10.864 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T18:52:10.864 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2023-12-19T18:52:10.865 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:52:10.866 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 96 M 2023-12-19T18:52:10.866 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 121 M 2023-12-19T18:52:10.866 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T18:52:10.937 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T18:52:10.937 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T18:52:13.620 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): kernel-core-5.14.0-391.el9.x86_64.rpm 7.5 MB/s | 20 MB 00:02 2023-12-19T18:52:13.730 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T18:52:14.388 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:52:14.526 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:52:16.020 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): kernel-5.14.0-391.el9.x86_64.rpm 1.1 MB/s | 5.6 MB 00:05 2023-12-19T18:52:16.554 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 11 MB/s | 32 MB 00:02 2023-12-19T18:52:17.425 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:52:17.633 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:52:18.312 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 5.1 MB/s | 38 MB 00:07 2023-12-19T18:52:18.312 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T18:52:18.312 INFO:teuthology.orchestra.run.smithi187.stdout:Total 13 MB/s | 96 MB 00:07 2023-12-19T18:52:19.165 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T18:52:19.525 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T18:52:19.525 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T18:52:21.120 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T18:52:21.121 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T18:52:22.622 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:52:22.748 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T18:52:24.394 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:52:24.520 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:52:27.248 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:52:27.430 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:52:27.762 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:52:32.271 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:36.914 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:58.569 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:58.613 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:59.002 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:59.002 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:52:59.002 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:52:59.002 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:52:59.436 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T18:52:59.904 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:53:00.059 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:53:00.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:53:00.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-19T18:53:00.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-19T18:53:00.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-19T18:53:00.060 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-19T18:53:00.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2023-12-19T18:53:00.060 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:53:00.183 INFO:teuthology.orchestra.run.smithi055.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:53:00.184 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:53:00.184 DEBUG:teuthology.orchestra.run.smithi055:> sudo rpm -qi grub2-tools 2023-12-19T18:53:00.222 INFO:teuthology.orchestra.run.smithi055.stdout:Name : grub2-tools 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Epoch : 1 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Version : 2.06 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Release : 68.el9 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: x86_64 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Group : Unspecified 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Size : 8264254 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:License : GPLv3+ 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Packager : builder@centos.org 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:Vendor : CentOS 2023-12-19T18:53:00.223 INFO:teuthology.orchestra.run.smithi055.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:Summary : Support tools for GRUB. 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:Description : 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:hardware devices. 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T18:53:00.224 INFO:teuthology.orchestra.run.smithi055.stdout:This subpackage provides tools for support of all platforms. 2023-12-19T18:53:00.225 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-19T18:53:00.225 INFO:teuthology.task.kernel:Updating grub on smithi055 to boot 5.14.0-391.el9.x86_64 2023-12-19T18:53:00.225 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-19T18:53:00.981 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2023-12-19T18:53:02.373 INFO:teuthology.orchestra.run.smithi055.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-19T18:53:02.413 INFO:teuthology.orchestra.run.smithi055.stderr:done 2023-12-19T18:53:02.418 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2023-12-19T18:53:02.434 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.c13mEQ5INZ 2023-12-19T18:53:02.435 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c13mEQ5INZ 2023-12-19T18:53:02.506 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.c13mEQ5INZ 2023-12-19T18:53:02.651 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.c13mEQ5INZ is 6KB 2023-12-19T18:53:02.699 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.c13mEQ5INZ 2023-12-19T18:53:02.715 DEBUG:teuthology.orchestra.run.smithi055:> sudo /bin/ls /boot/loader/entries || true 2023-12-19T18:53:02.783 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-19T18:53:02.783 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-19T18:53:02.783 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-19T18:53:02.783 INFO:teuthology.orchestra.run.smithi055.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-19T18:53:02.784 DEBUG:teuthology.orchestra.run.smithi055:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-19T18:53:02.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2023-12-19T18:53:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:53:05.511 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:53:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:53:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-19T18:53:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T18:53:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:53:06.360 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T18:53:06.856 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:53:07.004 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:53:07.004 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:53:07.004 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-19T18:53:07.004 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-19T18:53:07.004 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-19T18:53:07.004 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-19T18:53:07.005 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2023-12-19T18:53:07.005 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T18:53:07.122 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T18:53:07.123 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T18:53:07.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2023-12-19T18:53:07.156 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.06 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 68.el9 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Group : Unspecified 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 8264254 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-19T18:53:07.157 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : builder@centos.org 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T18:53:07.158 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2023-12-19T18:53:07.159 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-19T18:53:07.159 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 5.14.0-391.el9.x86_64 2023-12-19T18:53:07.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-19T18:53:07.797 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2023-12-19T18:53:09.136 INFO:teuthology.orchestra.run.smithi187.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-19T18:53:09.185 INFO:teuthology.orchestra.run.smithi187.stderr:done 2023-12-19T18:53:09.197 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2023-12-19T18:53:09.216 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.rYGMiM5rRK 2023-12-19T18:53:09.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rYGMiM5rRK 2023-12-19T18:53:09.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.rYGMiM5rRK 2023-12-19T18:53:09.425 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.rYGMiM5rRK is 6KB 2023-12-19T18:53:09.475 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.rYGMiM5rRK 2023-12-19T18:53:09.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2023-12-19T18:53:09.562 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-19T18:53:09.562 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-19T18:53:09.562 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-19T18:53:09.562 INFO:teuthology.orchestra.run.smithi187.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-19T18:53:09.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-19T18:53:09.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2023-12-19T18:53:32.945 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-19T18:53:32.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T18:53:32.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:53:39.693 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-19T18:53:39.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T18:53:39.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:53:51.506 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2023-12-19T18:53:58.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T18:54:00.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T18:54:00.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:54:07.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T18:54:07.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:54:18.897 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2023-12-19T18:54:30.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T18:54:30.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:54:33.685 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-12-19T18:54:34.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T18:54:34.126 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-391.el9.x86_64"... 2023-12-19T18:54:34.126 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-12-19T18:54:34.141 INFO:teuthology.orchestra.run.smithi055.stdout:5.14.0-391.el9.x86_64 2023-12-19T18:54:34.142 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-19T18:54:34.142 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T18:54:34.142 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-19T18:54:35.143 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-19T18:54:35.143 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T18:54:35.203 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2023-12-19T18:54:35.253 DEBUG:teuthology.parallel:result is None 2023-12-19T18:55:07.177 DEBUG:teuthology.orchestra.remote:timed out 2023-12-19T18:55:19.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T18:55:19.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T18:55:19.523 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-12-19T18:55:19.904 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T18:55:19.905 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-391.el9.x86_64"... 2023-12-19T18:55:19.905 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-12-19T18:55:19.921 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-391.el9.x86_64 2023-12-19T18:55:19.922 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-19T18:55:19.922 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T18:55:19.922 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-19T18:55:20.923 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-19T18:55:20.923 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T18:55:21.321 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2023-12-19T18:55:21.354 DEBUG:teuthology.parallel:result is None 2023-12-19T18:55:21.355 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-19T18:55:21.363 INFO:teuthology.task.internal:Creating test directory... 2023-12-19T18:55:21.363 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T18:55:21.367 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T18:55:21.385 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-19T18:55:21.392 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-19T18:55:21.400 INFO:teuthology.task.internal:Creating archive directory... 2023-12-19T18:55:21.400 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T18:55:21.428 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T18:55:21.455 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-19T18:55:21.462 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-19T18:55:21.462 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T18:55:21.491 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T18:55:21.539 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:55:21.542 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:55:21.553 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:55:21.560 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T18:55:21.562 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-19T18:55:21.569 INFO:teuthology.task.internal:Configuring sudo... 2023-12-19T18:55:21.569 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T18:55:21.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T18:55:21.602 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-19T18:55:21.611 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-19T18:55:21.612 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T18:55:21.645 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T18:55:21.661 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:55:21.745 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:55:21.796 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:55:21.873 DEBUG:teuthology.orchestra.run.smithi055:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:55:21.926 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T18:55:21.926 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T18:55:21.998 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:55:22.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T18:55:22.101 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:55:22.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T18:55:22.232 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T18:55:22.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T18:55:22.300 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2023-12-19T18:55:22.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2023-12-19T18:55:22.354 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T18:55:22.383 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T18:55:22.869 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-19T18:55:22.878 INFO:teuthology.task.internal:Starting timer... 2023-12-19T18:55:22.878 INFO:teuthology.run_tasks:Running task pcp... 2023-12-19T18:55:22.900 INFO:teuthology.run_tasks:Running task selinux... 2023-12-19T18:55:22.914 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-19T18:55:22.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo service auditd rotate 2023-12-19T18:55:22.990 INFO:teuthology.orchestra.run.smithi055.stdout:Rotating logs: 2023-12-19T18:55:22.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2023-12-19T18:55:23.052 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2023-12-19T18:55:23.054 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-19T18:55:23.054 DEBUG:teuthology.orchestra.run.smithi055:> /usr/sbin/getenforce 2023-12-19T18:55:23.075 INFO:teuthology.orchestra.run.smithi055.stdout:Permissive 2023-12-19T18:55:23.076 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2023-12-19T18:55:23.094 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2023-12-19T18:55:23.094 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi055.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2023-12-19T18:55:23.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T18:55:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:55:23.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T18:55:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T18:55:23.179 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-19T18:55:23.179 DEBUG:teuthology.orchestra.run.smithi055:> sudo /usr/sbin/setenforce permissive 2023-12-19T18:55:23.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2023-12-19T18:55:23.262 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-19T18:55:23.272 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-19T18:55:23.278 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-19T18:55:23.408 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-19T18:55:23.425 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-19T18:55:23.427 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-19T19:01:07.857 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2023-12-19T19:01:07.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T19:01:07.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:01:07.955 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-12-19T19:01:08.028 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-19T19:01:08.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:01:08.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:01:08.113 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-12-19T19:01:08.184 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:01:08.185 INFO:teuthology.run_tasks:Running task clock... 2023-12-19T19:01:08.208 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-19T19:01:08.208 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T19:01:08.208 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T19:01:08.211 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T19:01:08.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T19:01:08.243 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-19T19:01:08.255 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-19T19:01:08.266 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-19T19:01:08.272 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-19T19:01:08.299 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: ntpd: command not found 2023-12-19T19:01:08.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2023-12-19T19:01:08.315 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2023-12-19T19:01:08.316 INFO:teuthology.orchestra.run.smithi055.stdout:506 Cannot talk to daemon 2023-12-19T19:01:08.331 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-19T19:01:08.336 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-19T19:01:08.347 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-19T19:01:08.356 INFO:teuthology.orchestra.run.smithi055.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-19T19:01:08.390 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2023-12-19T19:01:08.393 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T19:01:08.393 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2023-12-19T19:01:08.393 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.394 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.394 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.394 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.403 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.407 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:01:08.408 INFO:teuthology.run_tasks:Running task install... 2023-12-19T19:01:08.417 DEBUG:teuthology.task.install:project ceph 2023-12-19T19:01:08.417 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-19T19:01:08.417 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-19T19:01:08.417 INFO:teuthology.task.install:Using flavor: default 2023-12-19T19:01:08.423 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-19T19:01:08.423 INFO:teuthology.task.install:extra packages: [] 2023-12-19T19:01:08.423 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-19T19:01:08.424 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:01:08.424 INFO:teuthology.packaging:ref: None 2023-12-19T19:01:08.424 INFO:teuthology.packaging:tag: None 2023-12-19T19:01:08.424 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:01:08.424 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:01:08.424 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T19:01:08.426 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-19T19:01:08.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:01:08.426 INFO:teuthology.packaging:ref: None 2023-12-19T19:01:08.426 INFO:teuthology.packaging:tag: None 2023-12-19T19:01:08.426 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:01:08.426 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:01:08.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T19:01:08.646 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/ 2023-12-19T19:01:08.647 INFO:teuthology.task.install.rpm:Package version is 17.2.7-131.ga88f3edf 2023-12-19T19:01:08.651 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/ 2023-12-19T19:01:08.651 INFO:teuthology.task.install.rpm:Package version is 17.2.7-131.ga88f3edf 2023-12-19T19:01:08.756 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-19T19:01:08.756 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:01:08.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-19T19:01:08.763 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/a88f3edf230e1c58be7443afb566f4b63b8b9c94/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-19T19:01:08.763 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:01:08.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-19T19:01:08.784 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-19T19:01:08.785 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:01:08.785 INFO:teuthology.packaging:ref: None 2023-12-19T19:01:08.785 INFO:teuthology.packaging:tag: None 2023-12-19T19:01:08.785 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:01:08.785 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:01:08.785 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-19T19:01:08.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-19T19:01:08.797 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:01:08.797 INFO:teuthology.packaging:ref: None 2023-12-19T19:01:08.797 INFO:teuthology.packaging:tag: None 2023-12-19T19:01:08.797 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:01:08.797 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:01:08.797 DEBUG:teuthology.orchestra.run.smithi055:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-19T19:01:08.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-19T19:01:08.869 DEBUG:teuthology.orchestra.run.smithi055:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-19T19:01:08.932 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-19T19:01:08.958 DEBUG:teuthology.orchestra.run.smithi055:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-19T19:01:08.987 INFO:teuthology.orchestra.run.smithi055.stdout:check_obsoletes = 1 2023-12-19T19:01:08.989 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum clean all 2023-12-19T19:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2023-12-19T19:01:09.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2023-12-19T19:01:09.243 INFO:teuthology.orchestra.run.smithi055.stdout:73 files removed 2023-12-19T19:01:09.244 INFO:teuthology.orchestra.run.smithi187.stdout:73 files removed 2023-12-19T19:01:09.270 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-radosgw 2023-12-19T19:01:09.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2023-12-19T19:01:09.943 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 220 kB/s | 77 kB 00:00 2023-12-19T19:01:10.008 INFO:teuthology.orchestra.run.smithi055.stdout:ceph packages for x86_64 192 kB/s | 77 kB 00:00 2023-12-19T19:01:10.311 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 29 kB/s | 9.1 kB 00:00 2023-12-19T19:01:10.442 INFO:teuthology.orchestra.run.smithi055.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2023-12-19T19:01:10.686 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 5.3 kB/s | 1.8 kB 00:00 2023-12-19T19:01:10.876 INFO:teuthology.orchestra.run.smithi055.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2023-12-19T19:01:11.138 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-19T19:01:11.303 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-19T19:01:12.472 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-12-19T19:01:12.687 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-12-19T19:01:13.536 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-12-19T19:01:15.438 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 7.9 MB 00:02 2023-12-19T19:01:16.671 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - AppStream 9.0 MB/s | 18 MB 00:02 2023-12-19T19:01:20.197 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 18 MB 00:03 2023-12-19T19:01:24.198 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 5.8 MB 00:00 2023-12-19T19:01:25.847 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 5.8 MB 00:05 2023-12-19T19:01:25.890 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2023-12-19T19:01:27.248 INFO:teuthology.orchestra.run.smithi055.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2023-12-19T19:01:27.250 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-12-19T19:01:27.900 INFO:teuthology.orchestra.run.smithi055.stdout:Extra Packages for Enterprise Linux 37 MB/s | 20 MB 00:00 2023-12-19T19:01:32.100 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-19T19:01:32.750 INFO:teuthology.orchestra.run.smithi055.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-12-19T19:01:34.578 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 14 M 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.1 M 2023-12-19T19:01:34.588 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 5.0 M 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 23 M 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 24 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 635 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 457 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 4.9 M 2023-12-19T19:01:34.589 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 45 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 116 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 156 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 319 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 292 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 99 k 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout:Install 22 Packages 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2023-12-19T19:01:34.590 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:01:34.591 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 58 M 2023-12-19T19:01:34.591 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:01:35.299 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 14 M 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrading: 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: librados2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.1 M 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 5.0 M 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 23 M 2023-12-19T19:01:35.317 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 24 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 635 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 457 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2 x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 4.9 M 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 45 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 116 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 156 k 2023-12-19T19:01:35.318 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 319 k 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 292 k 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 99 k 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout:Install 22 Packages 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout:Upgrade 2 Packages 2023-12-19T19:01:35.319 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:01:35.320 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 58 M 2023-12-19T19:01:35.320 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:01:39.584 INFO:teuthology.orchestra.run.smithi055.stdout:(1/24): ceph-base-17.2.7-131.ga88f3edf.el9.x86_ 2.3 MB/s | 5.0 MB 00:02 2023-12-19T19:01:39.660 INFO:teuthology.orchestra.run.smithi055.stdout:(2/24): ceph-selinux-17.2.7-131.ga88f3edf.el9.x 320 kB/s | 24 kB 00:00 2023-12-19T19:01:39.910 INFO:teuthology.orchestra.run.smithi055.stdout:(3/24): libcephfs2-17.2.7-131.ga88f3edf.el9.x86 2.5 MB/s | 635 kB 00:00 2023-12-19T19:01:40.093 INFO:teuthology.orchestra.run.smithi055.stdout:(4/24): libradosstriper1-17.2.7-131.ga88f3edf.e 2.4 MB/s | 457 kB 00:00 2023-12-19T19:01:41.398 INFO:teuthology.orchestra.run.smithi055.stdout:(5/24): librgw2-17.2.7-131.ga88f3edf.el9.x86_64 3.8 MB/s | 4.9 MB 00:01 2023-12-19T19:01:41.490 INFO:teuthology.orchestra.run.smithi055.stdout:(6/24): python3-ceph-argparse-17.2.7-131.ga88f3 494 kB/s | 45 kB 00:00 2023-12-19T19:01:41.582 INFO:teuthology.orchestra.run.smithi055.stdout:(7/24): python3-ceph-common-17.2.7-131.ga88f3ed 1.2 MB/s | 116 kB 00:00 2023-12-19T19:01:41.682 INFO:teuthology.orchestra.run.smithi055.stdout:(8/24): python3-cephfs-17.2.7-131.ga88f3edf.el9 1.5 MB/s | 156 kB 00:00 2023-12-19T19:01:41.841 INFO:teuthology.orchestra.run.smithi055.stdout:(9/24): python3-rados-17.2.7-131.ga88f3edf.el9. 2.0 MB/s | 319 kB 00:00 2023-12-19T19:01:41.942 INFO:teuthology.orchestra.run.smithi055.stdout:(10/24): python3-rbd-17.2.7-131.ga88f3edf.el9.x 2.9 MB/s | 292 kB 00:00 2023-12-19T19:01:42.042 INFO:teuthology.orchestra.run.smithi055.stdout:(11/24): python3-rgw-17.2.7-131.ga88f3edf.el9.x 996 kB/s | 99 kB 00:00 2023-12-19T19:01:42.159 INFO:teuthology.orchestra.run.smithi055.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-12-19T19:01:42.192 INFO:teuthology.orchestra.run.smithi055.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-12-19T19:01:42.251 INFO:teuthology.orchestra.run.smithi055.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-19T19:01:42.310 INFO:teuthology.orchestra.run.smithi055.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2023-12-19T19:01:42.343 INFO:teuthology.orchestra.run.smithi055.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2023-12-19T19:01:42.594 INFO:teuthology.orchestra.run.smithi055.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2023-12-19T19:01:42.653 INFO:teuthology.orchestra.run.smithi055.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2023-12-19T19:01:42.719 INFO:teuthology.orchestra.run.smithi055.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2023-12-19T19:01:42.878 INFO:teuthology.orchestra.run.smithi055.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2023-12-19T19:01:43.978 INFO:teuthology.orchestra.run.smithi055.stdout:(21/24): librados2-17.2.7-131.ga88f3edf.el9.x86 2.8 MB/s | 3.1 MB 00:01 2023-12-19T19:01:44.267 INFO:teuthology.orchestra.run.smithi187.stdout:(1/24): ceph-base-17.2.7-131.ga88f3edf.el9.x86_ 718 kB/s | 5.0 MB 00:07 2023-12-19T19:01:44.384 INFO:teuthology.orchestra.run.smithi187.stdout:(2/24): ceph-selinux-17.2.7-131.ga88f3edf.el9.x 206 kB/s | 24 kB 00:00 2023-12-19T19:01:45.042 INFO:teuthology.orchestra.run.smithi187.stdout:(3/24): libcephfs2-17.2.7-131.ga88f3edf.el9.x86 964 kB/s | 635 kB 00:00 2023-12-19T19:01:45.170 INFO:teuthology.orchestra.run.smithi055.stdout:(22/24): ceph-common-17.2.7-131.ga88f3edf.el9.x 2.9 MB/s | 23 MB 00:07 2023-12-19T19:01:45.237 INFO:teuthology.orchestra.run.smithi055.stdout:(23/24): librbd1-17.2.7-131.ga88f3edf.el9.x86_6 2.3 MB/s | 2.9 MB 00:01 2023-12-19T19:01:45.401 INFO:teuthology.orchestra.run.smithi187.stdout:(4/24): libradosstriper1-17.2.7-131.ga88f3edf.e 1.2 MB/s | 457 kB 00:00 2023-12-19T19:01:45.685 INFO:teuthology.orchestra.run.smithi187.stdout:(5/24): ceph-radosgw-17.2.7-131.ga88f3edf.el9.x 1.6 MB/s | 14 MB 00:08 2023-12-19T19:01:45.752 INFO:teuthology.orchestra.run.smithi187.stdout:(6/24): python3-ceph-argparse-17.2.7-131.ga88f3 680 kB/s | 45 kB 00:00 2023-12-19T19:01:45.835 INFO:teuthology.orchestra.run.smithi187.stdout:(7/24): python3-ceph-common-17.2.7-131.ga88f3ed 1.4 MB/s | 116 kB 00:00 2023-12-19T19:01:45.977 INFO:teuthology.orchestra.run.smithi187.stdout:(8/24): python3-cephfs-17.2.7-131.ga88f3edf.el9 1.1 MB/s | 156 kB 00:00 2023-12-19T19:01:46.161 INFO:teuthology.orchestra.run.smithi187.stdout:(9/24): python3-rados-17.2.7-131.ga88f3edf.el9. 1.7 MB/s | 319 kB 00:00 2023-12-19T19:01:46.311 INFO:teuthology.orchestra.run.smithi187.stdout:(10/24): python3-rbd-17.2.7-131.ga88f3edf.el9.x 1.9 MB/s | 292 kB 00:00 2023-12-19T19:01:46.395 INFO:teuthology.orchestra.run.smithi187.stdout:(11/24): python3-rgw-17.2.7-131.ga88f3edf.el9.x 1.2 MB/s | 99 kB 00:00 2023-12-19T19:01:46.512 INFO:teuthology.orchestra.run.smithi187.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-12-19T19:01:46.562 INFO:teuthology.orchestra.run.smithi187.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2023-12-19T19:01:46.621 INFO:teuthology.orchestra.run.smithi187.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-12-19T19:01:46.663 INFO:teuthology.orchestra.run.smithi187.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2023-12-19T19:01:46.688 INFO:teuthology.orchestra.run.smithi187.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2023-12-19T19:01:47.197 INFO:teuthology.orchestra.run.smithi187.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 608 kB/s | 309 kB 00:00 2023-12-19T19:01:47.305 INFO:teuthology.orchestra.run.smithi187.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2023-12-19T19:01:47.422 INFO:teuthology.orchestra.run.smithi187.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2023-12-19T19:01:47.781 INFO:teuthology.orchestra.run.smithi187.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2023-12-19T19:01:48.104 INFO:teuthology.orchestra.run.smithi055.stdout:(24/24): ceph-radosgw-17.2.7-131.ga88f3edf.el9. 1.3 MB/s | 14 MB 00:10 2023-12-19T19:01:48.107 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:01:48.108 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.5 MB/s | 58 MB 00:12 2023-12-19T19:01:48.108 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:01:48.146 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:01:48.146 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:01:48.636 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:01:48.636 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:01:48.781 INFO:teuthology.orchestra.run.smithi187.stdout:(21/24): librgw2-17.2.7-131.ga88f3edf.el9.x86_6 1.5 MB/s | 4.9 MB 00:03 2023-12-19T19:01:49.206 INFO:teuthology.orchestra.run.smithi187.stdout:(22/24): librados2-17.2.7-131.ga88f3edf.el9.x86 2.2 MB/s | 3.1 MB 00:01 2023-12-19T19:01:49.886 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:01:50.173 INFO:teuthology.orchestra.run.smithi187.stdout:(23/24): librbd1-17.2.7-131.ga88f3edf.el9.x86_6 2.1 MB/s | 2.9 MB 00:01 2023-12-19T19:01:50.255 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-19T19:01:50.323 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:01:50.415 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:01:50.505 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T19:01:50.533 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:01:50.605 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:01:50.658 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-19T19:01:50.705 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-19T19:01:51.162 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-19T19:01:51.186 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T19:01:51.240 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T19:01:51.270 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-19T19:01:51.305 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 10/26 2023-12-19T19:01:51.323 INFO:teuthology.orchestra.run.smithi187.stdout:(24/24): ceph-common-17.2.7-131.ga88f3edf.el9.x 1.6 MB/s | 23 MB 00:14 2023-12-19T19:01:51.326 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:01:51.327 INFO:teuthology.orchestra.run.smithi187.stdout:Total 3.5 MB/s | 58 MB 00:16 2023-12-19T19:01:51.327 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:01:51.334 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T19:01:51.361 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:01:51.361 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:01:51.397 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T19:01:51.421 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:01:51.487 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:01:51.528 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-19T19:01:51.571 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-19T19:01:51.603 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-19T19:01:51.839 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:01:51.839 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:01:51.926 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-19T19:01:51.950 INFO:teuthology.orchestra.run.smithi055.stdout: Upgrading : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T19:01:52.040 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T19:01:52.115 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 19/26 2023-12-19T19:01:52.144 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 20/26 2023-12-19T19:01:53.100 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:01:53.444 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-19T19:01:53.492 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:01:53.582 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:01:53.661 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T19:01:53.685 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:01:53.774 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:01:53.823 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-19T19:01:53.856 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-19T19:01:54.290 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-19T19:01:54.313 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T19:01:54.359 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 8/26 2023-12-19T19:01:54.387 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-19T19:01:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 10/26 2023-12-19T19:01:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T19:01:54.550 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T19:01:54.577 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:01:54.642 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:01:54.669 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-19T19:01:54.707 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-19T19:01:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-19T19:01:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-19T19:01:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T19:01:55.145 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 18/26 2023-12-19T19:01:55.218 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 19/26 2023-12-19T19:01:55.241 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 20/26 2023-12-19T19:01:55.258 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:55.294 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T19:01:55.366 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T19:01:55.366 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-19T19:01:55.366 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-19T19:01:55.366 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:01:55.403 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:01:57.615 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:57.663 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 21/26 2023-12-19T19:01:58.224 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T19:01:58.325 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 22/26 2023-12-19T19:01:58.325 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-19T19:01:58.325 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-19T19:01:58.325 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:01:58.368 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /sys 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /proc 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /mnt 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /var/tmp 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /home 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /root 2023-12-19T19:02:05.135 INFO:teuthology.orchestra.run.smithi055.stdout:skipping the directory /tmp 2023-12-19T19:02:05.136 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:06.445 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T19:02:06.524 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T19:02:06.525 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:06.525 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-19T19:02:06.525 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T19:02:06.525 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T19:02:06.525 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:06.526 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T19:02:06.614 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T19:02:06.615 INFO:teuthology.orchestra.run.smithi055.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2023-12-19T19:02:07.825 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:07.887 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 1/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 5/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 6/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 7/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 8/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 9/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 10/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T19:02:07.888 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-19T19:02:07.889 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 25/26 2023-12-19T19:02:08.885 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout:Upgraded: 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-19T19:02:08.886 INFO:teuthology.orchestra.run.smithi055.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:08.887 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:02:09.074 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T19:02:09.149 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 24/26 2023-12-19T19:02:09.150 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:09.150 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-19T19:02:09.150 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T19:02:09.150 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T19:02:09.150 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:09.151 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T19:02:09.243 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-test 2023-12-19T19:02:09.247 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-19T19:02:09.247 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:09.823 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:00:37 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:09.981 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:02:09.981 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 48 M 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 158 k 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:02:09.982 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:02:09.983 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-12-19T19:02:09.983 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:09.983 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 49 M 2023-12-19T19:02:09.983 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 195 M 2023-12-19T19:02:09.983 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:02:10.343 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 519 kB/s | 186 kB 00:00 2023-12-19T19:02:10.448 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 1/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 3/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 4/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 5/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 6/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 7/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x 8/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86 9/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 10/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 11/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 12/26 2023-12-19T19:02:10.449 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 13/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 23/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-19T19:02:10.450 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 25/26 2023-12-19T19:02:10.459 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2023-12-19T19:02:10.593 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): libcephsqlite-17.2.7-131.ga88f3edf.el9.x 260 kB/s | 158 kB 00:00 2023-12-19T19:02:10.627 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.8 MB/s | 300 kB 00:00 2023-12-19T19:02:10.644 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.336 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:11.337 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-19T19:02:11.338 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:11.338 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:11.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2023-12-19T19:02:12.209 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:40 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:12.352 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:12.352 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:12.352 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:02:12.352 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 48 M 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 158 k 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2023-12-19T19:02:12.353 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:12.354 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 49 M 2023-12-19T19:02:12.354 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 195 M 2023-12-19T19:02:12.354 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:12.635 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 665 kB/s | 186 kB 00:00 2023-12-19T19:02:12.727 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2023-12-19T19:02:12.827 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2023-12-19T19:02:12.886 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-12-19T19:02:13.161 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): libcephsqlite-17.2.7-131.ga88f3edf.el9.x 197 kB/s | 158 kB 00:00 2023-12-19T19:02:19.460 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): ceph-test-17.2.7-131.ga88f3edf.el9.x86_6 6.7 MB/s | 48 MB 00:07 2023-12-19T19:02:19.464 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:19.464 INFO:teuthology.orchestra.run.smithi187.stdout:Total 6.9 MB/s | 49 MB 00:07 2023-12-19T19:02:19.464 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:02:19.483 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:02:19.876 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:02:19.876 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:02:20.488 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:02:20.576 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-19T19:02:20.619 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-19T19:02:20.651 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-19T19:02:20.693 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-19T19:02:20.722 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T19:02:25.127 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T19:02:25.213 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T19:02:27.501 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T19:02:27.501 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 1/6 2023-12-19T19:02:27.502 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2/6 2023-12-19T19:02:27.502 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-19T19:02:27.502 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-19T19:02:27.502 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: jq-1.6-16.el9.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:27.902 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:28.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2023-12-19T19:02:28.773 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:56 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:28.926 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:28.928 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2023-12-19T19:02:28.928 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:02:28.928 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2023-12-19T19:02:28.928 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 6.5 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.1 M 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 1.4 M 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 238 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.7 M 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 15 M 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-19T19:02:28.929 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-19T19:02:28.930 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================== 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout:Install 33 Packages 2023-12-19T19:02:28.931 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:28.932 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 27 M 2023-12-19T19:02:28.932 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 99 M 2023-12-19T19:02:28.932 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:29.212 INFO:teuthology.orchestra.run.smithi187.stdout:(1/33): ceph-17.2.7-131.ga88f3edf.el9.x86_64.rp 29 kB/s | 6.5 kB 00:00 2023-12-19T19:02:32.337 INFO:teuthology.orchestra.run.smithi187.stdout:(2/33): ceph-mds-17.2.7-131.ga88f3edf.el9.x86_6 631 kB/s | 2.1 MB 00:03 2023-12-19T19:02:32.446 INFO:teuthology.orchestra.run.smithi187.stdout:(3/33): ceph-mgr-17.2.7-131.ga88f3edf.el9.x86_6 416 kB/s | 1.4 MB 00:03 2023-12-19T19:02:32.704 INFO:teuthology.orchestra.run.smithi187.stdout:(4/33): ceph-mgr-modules-core-17.2.7-131.ga88f3 924 kB/s | 238 kB 00:00 2023-12-19T19:02:32.846 INFO:teuthology.orchestra.run.smithi187.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2023-12-19T19:02:32.880 INFO:teuthology.orchestra.run.smithi187.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2023-12-19T19:02:32.922 INFO:teuthology.orchestra.run.smithi187.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2023-12-19T19:02:32.963 INFO:teuthology.orchestra.run.smithi187.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 932 kB/s | 39 kB 00:00 2023-12-19T19:02:33.022 INFO:teuthology.orchestra.run.smithi187.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-12-19T19:02:33.089 INFO:teuthology.orchestra.run.smithi187.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-19T19:02:33.123 INFO:teuthology.orchestra.run.smithi187.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-19T19:02:33.165 INFO:teuthology.orchestra.run.smithi187.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-12-19T19:02:33.198 INFO:teuthology.orchestra.run.smithi187.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-12-19T19:02:33.232 INFO:teuthology.orchestra.run.smithi187.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-12-19T19:02:33.366 INFO:teuthology.orchestra.run.smithi187.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 326 kB/s | 43 kB 00:00 2023-12-19T19:02:33.483 INFO:teuthology.orchestra.run.smithi187.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2023-12-19T19:02:33.575 INFO:teuthology.orchestra.run.smithi187.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2023-12-19T19:02:33.625 INFO:teuthology.orchestra.run.smithi187.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2023-12-19T19:02:33.667 INFO:teuthology.orchestra.run.smithi187.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2023-12-19T19:02:33.709 INFO:teuthology.orchestra.run.smithi187.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 555 kB/s | 23 kB 00:00 2023-12-19T19:02:33.750 INFO:teuthology.orchestra.run.smithi187.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 468 kB/s | 19 kB 00:00 2023-12-19T19:02:33.793 INFO:teuthology.orchestra.run.smithi187.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 470 kB/s | 20 kB 00:00 2023-12-19T19:02:33.834 INFO:teuthology.orchestra.run.smithi187.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-12-19T19:02:33.893 INFO:teuthology.orchestra.run.smithi187.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2023-12-19T19:02:33.960 INFO:teuthology.orchestra.run.smithi187.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2023-12-19T19:02:34.027 INFO:teuthology.orchestra.run.smithi187.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2023-12-19T19:02:34.086 INFO:teuthology.orchestra.run.smithi187.stdout:(27/33): ceph-mon-17.2.7-131.ga88f3edf.el9.x86_ 776 kB/s | 3.7 MB 00:04 2023-12-19T19:02:34.111 INFO:teuthology.orchestra.run.smithi187.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2023-12-19T19:02:34.178 INFO:teuthology.orchestra.run.smithi187.stdout:(29/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2023-12-19T19:02:34.203 INFO:teuthology.orchestra.run.smithi187.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2023-12-19T19:02:34.245 INFO:teuthology.orchestra.run.smithi187.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2023-12-19T19:02:34.270 INFO:teuthology.orchestra.run.smithi187.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2023-12-19T19:02:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:(33/33): ceph-osd-17.2.7-131.ga88f3edf.el9.x86_ 4.1 MB/s | 15 MB 00:03 2023-12-19T19:02:36.057 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:36.058 INFO:teuthology.orchestra.run.smithi187.stdout:Total 3.7 MB/s | 27 MB 00:07 2023-12-19T19:02:36.058 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:02:36.096 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:02:36.096 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:02:36.437 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:02:36.437 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:02:37.660 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:02:37.715 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-19T19:02:37.746 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-19T19:02:37.779 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-19T19:02:37.870 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-19T19:02:37.941 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-19T19:02:37.982 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-19T19:02:38.053 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-19T19:02:38.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-19T19:02:38.146 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-19T19:02:38.195 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-19T19:02:38.250 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-19T19:02:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-19T19:02:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-19T19:02:38.534 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-19T19:02:38.573 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-19T19:02:38.603 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-19T19:02:38.679 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-19T19:02:38.722 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T19:02:38.755 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-19T19:02:38.796 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-19T19:02:38.830 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-19T19:02:38.857 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-19T19:02:38.945 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-19T19:02:38.984 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-19T19:02:39.051 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-19T19:02:39.187 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-19T19:02:39.313 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T19:02:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 28/33 2023-12-19T19:02:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T19:02:39.527 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T19:02:39.527 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:39.527 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-19T19:02:39.527 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T19:02:39.527 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T19:02:39.528 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:40.868 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T19:02:40.895 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:41.294 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T19:02:41.318 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:41.520 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T19:02:41.543 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T19:02:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-19T19:02:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T19:02:41.544 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T19:02:41.544 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:41.601 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T19:02:43.293 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 1/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 3/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 4/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 5/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 6/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-19T19:02:43.294 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-19T19:02:43.295 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-19T19:02:43.296 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:44.229 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-19T19:02:44.230 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:44.231 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:44.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2023-12-19T19:02:45.181 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:13 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:45.274 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:02:45.323 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:45.323 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:02:45.324 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:45.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2023-12-19T19:02:45.740 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): ceph-test-17.2.7-131.ga88f3edf.el9.x86_6 1.3 MB/s | 48 MB 00:35 2023-12-19T19:02:45.743 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:45.743 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.4 MB/s | 49 MB 00:35 2023-12-19T19:02:45.743 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:02:45.762 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:02:45.762 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:02:45.931 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:13 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:46.073 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 80 k 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:46.074 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:46.075 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:02:46.075 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:46.075 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 80 k 2023-12-19T19:02:46.075 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 364 k 2023-12-19T19:02:46.075 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:46.172 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:02:46.172 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:02:46.628 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-17.2.7-131.ga88f3edf.el9.noarch.rpm 146 kB/s | 80 kB 00:00 2023-12-19T19:02:46.628 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:46.628 INFO:teuthology.orchestra.run.smithi187.stdout:Total 145 kB/s | 80 kB 00:00 2023-12-19T19:02:46.629 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:02:46.634 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:02:46.634 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:02:46.636 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:02:46.636 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:02:46.718 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:02:46.771 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:02:46.851 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-19T19:02:46.894 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-19T19:02:46.923 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-19T19:02:46.966 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-19T19:02:46.994 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T19:02:47.117 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:02:47.208 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:02:47.592 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:02:47.898 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:02:47.898 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:47.898 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:47.898 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:02:47.899 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:47.899 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:48.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2023-12-19T19:02:48.670 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:16 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:48.816 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:48.817 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:48.817 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-12-19T19:02:48.817 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 137 k 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 137 k 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 378 k 2023-12-19T19:02:48.818 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:49.384 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-17.2.7-131.ga88f3ed 242 kB/s | 137 kB 00:00 2023-12-19T19:02:49.384 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:49.384 INFO:teuthology.orchestra.run.smithi187.stdout:Total 241 kB/s | 137 kB 00:00 2023-12-19T19:02:49.384 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:02:49.392 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:02:49.392 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:02:49.444 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:02:49.445 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:02:49.595 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:02:49.643 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:02:49.667 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:02:49.667 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:02:49.668 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-19T19:02:49.668 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:50.666 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:50.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2023-12-19T19:02:51.458 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:19 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:51.552 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:02:51.601 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:51.602 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:02:51.602 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:51.633 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 5/6 2023-12-19T19:02:51.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-dashboard 2023-12-19T19:02:51.711 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T19:02:52.218 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:20 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:52.362 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 1.7 M 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 23 k 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 15 k 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-19T19:02:52.363 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:Install 7 Packages 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.0 M 2023-12-19T19:02:52.364 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 8.1 M 2023-12-19T19:02:52.365 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:52.978 INFO:teuthology.orchestra.run.smithi187.stdout:(1/7): ceph-prometheus-alerts-17.2.7-131.ga88f3 28 kB/s | 15 kB 00:00 2023-12-19T19:02:52.995 INFO:teuthology.orchestra.run.smithi187.stdout:(2/7): ceph-grafana-dashboards-17.2.7-131.ga88f 42 kB/s | 23 kB 00:00 2023-12-19T19:02:53.187 INFO:teuthology.orchestra.run.smithi187.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 43 kB/s | 9.0 kB 00:00 2023-12-19T19:02:53.354 INFO:teuthology.orchestra.run.smithi187.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 114 kB/s | 41 kB 00:00 2023-12-19T19:02:53.379 INFO:teuthology.orchestra.run.smithi187.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2023-12-19T19:02:53.629 INFO:teuthology.orchestra.run.smithi187.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 685 kB/s | 188 kB 00:00 2023-12-19T19:02:53.830 INFO:teuthology.orchestra.run.smithi187.stdout:(7/7): ceph-mgr-dashboard-17.2.7-131.ga88f3edf. 1.2 MB/s | 1.7 MB 00:01 2023-12-19T19:02:53.832 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:02:53.832 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-19T19:02:53.832 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:02:53.841 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:02:53.841 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:02:53.914 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:02:53.915 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:02:53.956 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 6/6 2023-12-19T19:02:53.956 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 1/6 2023-12-19T19:02:53.956 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2/6 2023-12-19T19:02:53.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-19T19:02:53.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-19T19:02:53.957 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-19T19:02:54.216 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:02:54.297 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-19T19:02:54.331 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-test-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: jq-1.6-16.el9.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: libcephsqlite-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:54.379 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:02:54.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-19T19:02:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-19T19:02:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 5/7 2023-12-19T19:02:54.667 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph 2023-12-19T19:02:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 6/7 2023-12-19T19:02:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T19:02:55.245 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:23 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:55.263 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T19:02:55.263 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 1/7 2023-12-19T19:02:55.264 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2/7 2023-12-19T19:02:55.264 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 3/7 2023-12-19T19:02:55.264 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-19T19:02:55.264 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-19T19:02:55.264 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-19T19:02:55.396 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: ceph x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 6.5 k 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.1 M 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 1.4 M 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 238 k 2023-12-19T19:02:55.399 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 3.7 M 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 15 M 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-19T19:02:55.400 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:02:55.401 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================== 2023-12-19T19:02:55.402 INFO:teuthology.orchestra.run.smithi055.stdout:Install 33 Packages 2023-12-19T19:02:55.402 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:02:55.402 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 27 M 2023-12-19T19:02:55.403 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 99 M 2023-12-19T19:02:55.403 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:02:55.589 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-19T19:02:55.589 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:55.589 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:02:55.589 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:02:55.589 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:55.590 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:02:55.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-19T19:02:56.051 INFO:teuthology.orchestra.run.smithi055.stdout:(1/33): ceph-17.2.7-131.ga88f3edf.el9.x86_64.rp 23 kB/s | 6.5 kB 00:00 2023-12-19T19:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:24 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:02:56.537 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 7.4 M 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-19T19:02:56.538 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout:=========================================================================================== 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout:Install 12 Packages 2023-12-19T19:02:56.539 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:02:56.540 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 37 M 2023-12-19T19:02:56.540 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 209 M 2023-12-19T19:02:56.540 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:02:56.937 INFO:teuthology.orchestra.run.smithi187.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 111 kB/s | 32 kB 00:00 2023-12-19T19:02:57.004 INFO:teuthology.orchestra.run.smithi187.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2023-12-19T19:02:57.105 INFO:teuthology.orchestra.run.smithi187.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 342 kB/s | 34 kB 00:00 2023-12-19T19:02:57.321 INFO:teuthology.orchestra.run.smithi187.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2023-12-19T19:02:57.763 INFO:teuthology.orchestra.run.smithi187.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.2 MB/s | 4.7 MB 00:00 2023-12-19T19:02:57.830 INFO:teuthology.orchestra.run.smithi187.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 4.8 MB 00:00 2023-12-19T19:02:57.855 INFO:teuthology.orchestra.run.smithi187.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-19T19:02:57.998 INFO:teuthology.orchestra.run.smithi187.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.6 MB/s | 803 kB 00:00 2023-12-19T19:02:58.018 INFO:teuthology.orchestra.run.smithi055.stdout:(2/33): ceph-mon-17.2.7-131.ga88f3edf.el9.x86_6 1.9 MB/s | 3.7 MB 00:01 2023-12-19T19:02:58.214 INFO:teuthology.orchestra.run.smithi187.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2023-12-19T19:02:58.239 INFO:teuthology.orchestra.run.smithi187.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 814 kB/s | 195 kB 00:00 2023-12-19T19:02:58.277 INFO:teuthology.orchestra.run.smithi055.stdout:(3/33): ceph-mds-17.2.7-131.ga88f3edf.el9.x86_6 842 kB/s | 2.1 MB 00:02 2023-12-19T19:02:58.319 INFO:teuthology.orchestra.run.smithi055.stdout:(4/33): ceph-mgr-17.2.7-131.ga88f3edf.el9.x86_6 563 kB/s | 1.4 MB 00:02 2023-12-19T19:02:58.435 INFO:teuthology.orchestra.run.smithi055.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2023-12-19T19:02:58.461 INFO:teuthology.orchestra.run.smithi055.stdout:(6/33): ceph-mgr-modules-core-17.2.7-131.ga88f3 1.3 MB/s | 238 kB 00:00 2023-12-19T19:02:58.486 INFO:teuthology.orchestra.run.smithi055.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2023-12-19T19:02:58.490 INFO:teuthology.orchestra.run.smithi187.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 894 kB/s | 245 kB 00:00 2023-12-19T19:02:58.511 INFO:teuthology.orchestra.run.smithi055.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2023-12-19T19:02:58.536 INFO:teuthology.orchestra.run.smithi055.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2023-12-19T19:02:58.562 INFO:teuthology.orchestra.run.smithi055.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-19T19:02:58.620 INFO:teuthology.orchestra.run.smithi055.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-12-19T19:02:58.646 INFO:teuthology.orchestra.run.smithi055.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-19T19:02:58.671 INFO:teuthology.orchestra.run.smithi055.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-19T19:02:58.696 INFO:teuthology.orchestra.run.smithi055.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-19T19:02:58.722 INFO:teuthology.orchestra.run.smithi055.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-12-19T19:02:58.997 INFO:teuthology.orchestra.run.smithi055.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2023-12-19T19:02:59.132 INFO:teuthology.orchestra.run.smithi055.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 420 kB/s | 172 kB 00:00 2023-12-19T19:02:59.206 INFO:teuthology.orchestra.run.smithi055.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2023-12-19T19:02:59.240 INFO:teuthology.orchestra.run.smithi055.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2023-12-19T19:02:59.282 INFO:teuthology.orchestra.run.smithi055.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 232 kB/s | 18 kB 00:00 2023-12-19T19:02:59.316 INFO:teuthology.orchestra.run.smithi055.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2023-12-19T19:02:59.358 INFO:teuthology.orchestra.run.smithi055.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2023-12-19T19:02:59.392 INFO:teuthology.orchestra.run.smithi055.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2023-12-19T19:02:59.442 INFO:teuthology.orchestra.run.smithi055.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2023-12-19T19:02:59.467 INFO:teuthology.orchestra.run.smithi055.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-19T19:02:59.543 INFO:teuthology.orchestra.run.smithi055.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2023-12-19T19:02:59.568 INFO:teuthology.orchestra.run.smithi055.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-12-19T19:02:59.626 INFO:teuthology.orchestra.run.smithi055.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2023-12-19T19:02:59.651 INFO:teuthology.orchestra.run.smithi055.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2023-12-19T19:02:59.727 INFO:teuthology.orchestra.run.smithi055.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-19T19:02:59.760 INFO:teuthology.orchestra.run.smithi055.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2023-12-19T19:02:59.802 INFO:teuthology.orchestra.run.smithi055.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2023-12-19T19:03:00.178 INFO:teuthology.orchestra.run.smithi055.stdout:(33/33): ceph-osd-17.2.7-131.ga88f3edf.el9.x86_ 7.1 MB/s | 15 MB 00:02 2023-12-19T19:03:00.183 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:00.184 INFO:teuthology.orchestra.run.smithi055.stdout:Total 5.6 MB/s | 27 MB 00:04 2023-12-19T19:03:00.184 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:00.227 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:00.227 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:00.579 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:00.580 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:01.290 INFO:teuthology.orchestra.run.smithi187.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 1.6 MB/s | 7.4 MB 00:04 2023-12-19T19:03:01.293 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:01.293 INFO:teuthology.orchestra.run.smithi187.stdout:Total 7.8 MB/s | 37 MB 00:04 2023-12-19T19:03:01.307 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:01.334 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:01.334 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:01.754 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:01.754 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:01.860 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:01.931 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-19T19:03:01.963 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-19T19:03:02.004 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-19T19:03:02.100 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-19T19:03:02.178 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-19T19:03:02.225 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-19T19:03:02.299 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-19T19:03:02.359 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-19T19:03:02.396 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-19T19:03:02.454 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:02.454 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-19T19:03:02.519 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-19T19:03:02.520 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-19T19:03:02.552 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-19T19:03:02.654 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-19T19:03:02.689 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-19T19:03:02.712 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-19T19:03:02.767 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-19T19:03:02.812 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-19T19:03:02.849 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-19T19:03:02.867 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T19:03:02.879 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-19T19:03:02.896 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T19:03:02.959 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-19T19:03:02.981 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-19T19:03:02.989 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T19:03:03.022 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-19T19:03:03.063 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-19T19:03:03.115 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-19T19:03:03.150 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-19T19:03:03.242 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-19T19:03:03.285 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-19T19:03:03.364 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-19T19:03:03.506 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-19T19:03:03.636 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-19T19:03:03.637 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T19:03:03.741 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-19T19:03:03.794 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 28/33 2023-12-19T19:03:03.818 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T19:03:03.845 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 29/33 2023-12-19T19:03:03.846 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:03.846 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-19T19:03:03.846 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T19:03:03.846 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T19:03:03.846 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:05.004 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-19T19:03:05.252 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 30/33 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T19:03:05.281 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:05.681 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 31/33 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T19:03:05.704 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:05.912 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 32/33 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T19:03:05.937 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:06.005 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T19:03:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-19T19:03:06.061 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 33/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 1/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 3/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 4/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 5/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.n 6/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-19T19:03:07.700 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-19T19:03:07.701 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-19T19:03:07.702 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-19T19:03:07.703 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-19T19:03:07.703 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-19T19:03:07.703 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-19T19:03:07.703 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-19T19:03:07.703 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-19T19:03:08.525 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-19T19:03:08.525 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:08.525 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:08.525 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:08.525 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mds-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mon-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-osd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-19T19:03:08.526 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ply-3.11-14.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-19T19:03:08.527 INFO:teuthology.orchestra.run.smithi055.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-19T19:03:08.528 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-19T19:03:08.528 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-19T19:03:08.528 INFO:teuthology.orchestra.run.smithi055.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-19T19:03:08.528 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:08.528 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 1/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-19T19:03:08.592 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-19T19:03:08.593 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-19T19:03:08.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-base 2023-12-19T19:03:09.005 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-19T19:03:09.005 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:09.006 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:09.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-rook 2023-12-19T19:03:09.485 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:37 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:09.582 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-base-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:09.634 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:09.635 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:09.635 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:09.697 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install cephadm 2023-12-19T19:03:09.949 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:37 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:10.094 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 47 k 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-19T19:03:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout:Install 13 Packages 2023-12-19T19:03:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:10.097 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.0 M 2023-12-19T19:03:10.097 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 26 M 2023-12-19T19:03:10.097 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:10.280 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:38 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:10.428 INFO:teuthology.orchestra.run.smithi187.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 225 kB/s | 24 kB 00:00 2023-12-19T19:03:10.428 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 80 k 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 80 k 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 364 k 2023-12-19T19:03:10.429 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:10.478 INFO:teuthology.orchestra.run.smithi187.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2023-12-19T19:03:10.536 INFO:teuthology.orchestra.run.smithi187.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2023-12-19T19:03:10.561 INFO:teuthology.orchestra.run.smithi187.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-12-19T19:03:10.586 INFO:teuthology.orchestra.run.smithi187.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-19T19:03:10.612 INFO:teuthology.orchestra.run.smithi187.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2023-12-19T19:03:10.912 INFO:teuthology.orchestra.run.smithi187.stdout:(7/13): ceph-mgr-rook-17.2.7-131.ga88f3edf.el9. 80 kB/s | 47 kB 00:00 2023-12-19T19:03:10.979 INFO:teuthology.orchestra.run.smithi187.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 82 kB/s | 32 kB 00:00 2023-12-19T19:03:11.005 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm-17.2.7-131.ga88f3edf.el9.noarch.rpm 140 kB/s | 80 kB 00:00 2023-12-19T19:03:11.005 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:11.005 INFO:teuthology.orchestra.run.smithi055.stdout:Total 140 kB/s | 80 kB 00:00 2023-12-19T19:03:11.005 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:11.006 INFO:teuthology.orchestra.run.smithi187.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2023-12-19T19:03:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:11.009 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:11.011 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:11.012 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:11.082 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:11.205 INFO:teuthology.orchestra.run.smithi187.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 334 kB/s | 59 kB 00:00 2023-12-19T19:03:11.388 INFO:teuthology.orchestra.run.smithi187.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2023-12-19T19:03:11.422 INFO:teuthology.orchestra.run.smithi187.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2023-12-19T19:03:11.470 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:03:11.522 INFO:teuthology.orchestra.run.smithi187.stdout:(13/13): python3-google-auth-2.23.4-1.el9.noarc 324 kB/s | 197 kB 00:00 2023-12-19T19:03:11.526 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:11.527 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-19T19:03:11.527 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:11.540 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:11.540 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:11.552 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:03:11.637 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:11.638 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:11.948 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:03:12.175 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:12.262 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/1 2023-12-19T19:03:12.262 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:12.262 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:12.263 INFO:teuthology.orchestra.run.smithi055.stdout: cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:12.263 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:12.263 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:12.337 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-19T19:03:12.398 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-19T19:03:12.453 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-19T19:03:12.465 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-immutable-object-cache 2023-12-19T19:03:12.482 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-19T19:03:12.524 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-19T19:03:12.616 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-19T19:03:12.717 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-19T19:03:12.766 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-19T19:03:13.046 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:41 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:13.197 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 137 k 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:13.198 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:13.199 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:03:13.199 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:13.199 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 137 k 2023-12-19T19:03:13.199 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 378 k 2023-12-19T19:03:13.199 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:13.485 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-19T19:03:13.539 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-19T19:03:13.572 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-19T19:03:13.608 INFO:teuthology.orchestra.run.smithi055.stdout:ceph-immutable-object-cache-17.2.7-131.ga88f3ed 335 kB/s | 137 kB 00:00 2023-12-19T19:03:13.609 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:13.609 INFO:teuthology.orchestra.run.smithi055.stdout:Total 334 kB/s | 137 kB 00:00 2023-12-19T19:03:13.609 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:13.617 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:13.618 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:13.618 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-19T19:03:13.642 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T19:03:13.674 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:13.674 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:13.819 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:13.892 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:03:13.917 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:03:13.917 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:13.917 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-19T19:03:13.917 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:14.399 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T19:03:14.399 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 1/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-19T19:03:14.400 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-19T19:03:14.793 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-19T19:03:14.794 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-19T19:03:14.795 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-19T19:03:14.795 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:14.795 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.e 1/1 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-immutable-object-cache-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:14.923 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:15.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr-cephadm 2023-12-19T19:03:15.154 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr 2023-12-19T19:03:15.577 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:43 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:15.725 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:15.725 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:15.725 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 121 k 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout:Install 6 Packages 2023-12-19T19:03:15.726 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:15.727 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 6.8 M 2023-12-19T19:03:15.727 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 33 M 2023-12-19T19:03:15.727 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:15.735 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:43 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:15.831 INFO:teuthology.orchestra.run.smithi055.stdout:Package ceph-mgr-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:15.884 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:15.885 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:15.885 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:15.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-dashboard 2023-12-19T19:03:16.073 INFO:teuthology.orchestra.run.smithi187.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-12-19T19:03:16.207 INFO:teuthology.orchestra.run.smithi187.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-12-19T19:03:16.524 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:44 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:16.549 INFO:teuthology.orchestra.run.smithi187.stdout:(3/6): ceph-mgr-cephadm-17.2.7-131.ga88f3edf.el 186 kB/s | 121 kB 00:00 2023-12-19T19:03:16.674 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 1.7 M 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 23 k 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 15 k 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-19T19:03:16.675 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout:Installing weak dependencies: 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout:Install 7 Packages 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:16.676 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2023-12-19T19:03:16.677 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 8.1 M 2023-12-19T19:03:16.677 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:17.174 INFO:teuthology.orchestra.run.smithi187.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 60 kB/s | 58 kB 00:00 2023-12-19T19:03:17.186 INFO:teuthology.orchestra.run.smithi055.stdout:(1/7): ceph-grafana-dashboards-17.2.7-131.ga88f 54 kB/s | 23 kB 00:00 2023-12-19T19:03:17.199 INFO:teuthology.orchestra.run.smithi187.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 78 kB/s | 51 kB 00:00 2023-12-19T19:03:17.212 INFO:teuthology.orchestra.run.smithi055.stdout:(2/7): ceph-prometheus-alerts-17.2.7-131.ga88f3 32 kB/s | 15 kB 00:00 2023-12-19T19:03:17.379 INFO:teuthology.orchestra.run.smithi055.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 47 kB/s | 9.0 kB 00:00 2023-12-19T19:03:17.382 INFO:teuthology.orchestra.run.smithi187.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 419 kB/s | 548 kB 00:01 2023-12-19T19:03:17.385 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:17.385 INFO:teuthology.orchestra.run.smithi187.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2023-12-19T19:03:17.385 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:17.396 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:17.396 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:17.489 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:17.489 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:17.538 INFO:teuthology.orchestra.run.smithi055.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 126 kB/s | 41 kB 00:00 2023-12-19T19:03:17.579 INFO:teuthology.orchestra.run.smithi055.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 154 kB/s | 31 kB 00:00 2023-12-19T19:03:17.783 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:17.804 INFO:teuthology.orchestra.run.smithi055.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2023-12-19T19:03:17.947 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-19T19:03:18.001 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-19T19:03:18.130 INFO:teuthology.orchestra.run.smithi055.stdout:(7/7): ceph-mgr-dashboard-17.2.7-131.ga88f3edf. 1.2 MB/s | 1.7 MB 00:01 2023-12-19T19:03:18.132 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:18.132 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-19T19:03:18.133 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:18.142 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:18.142 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:18.217 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:18.217 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:18.511 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-19T19:03:18.529 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:18.581 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-19T19:03:18.603 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-19T19:03:18.616 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-19T19:03:18.658 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-19T19:03:18.663 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-19T19:03:18.688 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T19:03:18.702 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-19T19:03:18.731 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 5/7 2023-12-19T19:03:18.971 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 6/7 2023-12-19T19:03:19.001 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T19:03:19.526 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T19:03:19.527 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/6 2023-12-19T19:03:19.527 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-19T19:03:19.527 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-19T19:03:19.527 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-19T19:03:19.527 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-19T19:03:19.569 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 7/7 2023-12-19T19:03:19.569 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.n 1/7 2023-12-19T19:03:19.569 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2/7 2023-12-19T19:03:19.569 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.no 3/7 2023-12-19T19:03:19.570 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-19T19:03:19.570 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-19T19:03:19.570 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-19T19:03:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-19T19:03:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:19.834 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:19.835 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-grafana-dashboards-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-prometheus-alerts-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:19.896 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-19T19:03:19.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-19T19:03:19.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-19T19:03:19.897 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-19T19:03:19.897 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:19.897 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:20.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2023-12-19T19:03:20.099 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-19T19:03:20.653 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:48 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:20.685 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:01:48 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:20.800 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 810 k 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2023-12-19T19:03:20.801 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:20.802 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 889 k 2023-12-19T19:03:20.802 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 2.6 M 2023-12-19T19:03:20.802 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:20.836 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 7.4 M 2023-12-19T19:03:20.837 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout:=========================================================================================== 2023-12-19T19:03:20.838 INFO:teuthology.orchestra.run.smithi055.stdout:Install 12 Packages 2023-12-19T19:03:20.839 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:20.839 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 37 M 2023-12-19T19:03:20.839 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 209 M 2023-12-19T19:03:20.839 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:21.025 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 353 kB/s | 79 kB 00:00 2023-12-19T19:03:21.078 INFO:teuthology.orchestra.run.smithi055.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 178 kB/s | 32 kB 00:00 2023-12-19T19:03:21.120 INFO:teuthology.orchestra.run.smithi055.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 413 kB/s | 17 kB 00:00 2023-12-19T19:03:21.171 INFO:teuthology.orchestra.run.smithi055.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2023-12-19T19:03:21.254 INFO:teuthology.orchestra.run.smithi055.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2023-12-19T19:03:21.592 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-fuse-17.2.7-131.ga88f3edf.el9.x86_6 1.0 MB/s | 810 kB 00:00 2023-12-19T19:03:21.593 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:21.594 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.1 MB/s | 889 kB 00:00 2023-12-19T19:03:21.594 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:21.616 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:21.617 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:21.696 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:21.696 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:21.891 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:22.037 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-19T19:03:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T19:03:22.770 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T19:03:22.770 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/2 2023-12-19T19:03:23.021 INFO:teuthology.orchestra.run.smithi055.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 2.6 MB/s | 4.7 MB 00:01 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:23.093 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:23.113 INFO:teuthology.orchestra.run.smithi055.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.6 MB/s | 4.8 MB 00:01 2023-12-19T19:03:23.138 INFO:teuthology.orchestra.run.smithi055.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-12-19T19:03:23.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2023-12-19T19:03:23.464 INFO:teuthology.orchestra.run.smithi055.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2023-12-19T19:03:23.781 INFO:teuthology.orchestra.run.smithi055.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2023-12-19T19:03:23.805 INFO:teuthology.orchestra.run.smithi055.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 575 kB/s | 195 kB 00:00 2023-12-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:51 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:24.062 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:24.063 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:24.063 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-12-19T19:03:24.063 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:24.063 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:24.063 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 126 k 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 126 k 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 448 k 2023-12-19T19:03:24.064 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:24.356 INFO:teuthology.orchestra.run.smithi055.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2023-12-19T19:03:24.823 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-17.2.7-131.ga88f3edf.el9.x86_64. 166 kB/s | 126 kB 00:00 2023-12-19T19:03:24.824 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:24.824 INFO:teuthology.orchestra.run.smithi187.stdout:Total 166 kB/s | 126 kB 00:00 2023-12-19T19:03:24.824 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:24.830 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:24.830 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:24.888 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:24.888 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:25.036 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:25.148 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:25.969 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:25.969 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:26.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2023-12-19T19:03:26.581 INFO:teuthology.orchestra.run.smithi055.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 1.3 MB/s | 7.4 MB 00:05 2023-12-19T19:03:26.584 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:26.584 INFO:teuthology.orchestra.run.smithi055.stdout:Total 6.5 MB/s | 37 MB 00:05 2023-12-19T19:03:26.599 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:26.628 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:26.628 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:26.765 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:54 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:26.858 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:26.908 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:26.909 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:26.909 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:26.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2023-12-19T19:03:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:27.057 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:27.539 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:55 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:27.686 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 25 k 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 25 k 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 110 k 2023-12-19T19:03:27.687 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:27.876 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-17.2.7-131.ga88f3edf.el9.x86_64 132 kB/s | 25 kB 00:00 2023-12-19T19:03:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:Total 131 kB/s | 25 kB 00:00 2023-12-19T19:03:27.877 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:27.881 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:27.882 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:27.883 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:27.908 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:27.973 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:27.979 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-19T19:03:28.010 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-19T19:03:28.083 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:28.109 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-19T19:03:28.155 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-19T19:03:28.340 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T19:03:28.371 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T19:03:28.452 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-19T19:03:28.474 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:28.755 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:28.755 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:28.755 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:28.755 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:28.755 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:28.756 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:28.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2023-12-19T19:03:29.152 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-19T19:03:29.279 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-19T19:03:29.487 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:57 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:29.582 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:29.632 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:29.633 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:29.633 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:29.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2023-12-19T19:03:30.246 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:58 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:30.340 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:30.390 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:30.391 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:30.391 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:30.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2023-12-19T19:03:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-19T19:03:31.021 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:59 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:31.115 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:31.166 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:31.167 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:31.167 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:31.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2023-12-19T19:03:31.675 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-19T19:03:31.704 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T19:03:31.798 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:59 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:31.894 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:31.944 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:31.945 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:31.945 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:32.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2023-12-19T19:03:32.580 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:00 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:32.679 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:32.730 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:32.731 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:32.732 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:32.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2023-12-19T19:03:33.354 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:01 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:33.447 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:33.498 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:33.499 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2023-12-19T19:03:33.499 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:33.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2023-12-19T19:03:34.113 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:02 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:34.258 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:34.278 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 81 k 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 81 k 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 222 k 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:34.279 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 12/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3e 1/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T19:03:34.280 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-19T19:03:34.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-19T19:03:34.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-19T19:03:34.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-19T19:03:34.285 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-diskprediction-local-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-19T19:03:34.847 INFO:teuthology.orchestra.run.smithi055.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-19T19:03:34.848 INFO:teuthology.orchestra.run.smithi055.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-19T19:03:34.849 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:34.849 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:34.938 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-17.2.7-131.ga88f3edf.el9.x86_64.rpm 120 kB/s | 81 kB 00:00 2023-12-19T19:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout:Total 120 kB/s | 81 kB 00:00 2023-12-19T19:03:34.939 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:34.947 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:34.947 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:35.005 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:35.006 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:35.145 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:35.211 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-rook 2023-12-19T19:03:35.262 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:35.650 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:35.800 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:03 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:35.951 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:35.952 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 47 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-19T19:03:35.953 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout:Install 13 Packages 2023-12-19T19:03:35.954 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:35.955 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.0 M 2023-12-19T19:03:35.955 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 26 M 2023-12-19T19:03:35.955 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:35.971 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:35.972 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:35.972 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:35.972 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:36.115 INFO:teuthology.orchestra.run.smithi055.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 230 kB/s | 24 kB 00:00 2023-12-19T19:03:36.165 INFO:teuthology.orchestra.run.smithi055.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2023-12-19T19:03:36.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2023-12-19T19:03:36.215 INFO:teuthology.orchestra.run.smithi055.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2023-12-19T19:03:36.249 INFO:teuthology.orchestra.run.smithi055.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-12-19T19:03:36.274 INFO:teuthology.orchestra.run.smithi055.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-12-19T19:03:36.299 INFO:teuthology.orchestra.run.smithi055.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2023-12-19T19:03:36.542 INFO:teuthology.orchestra.run.smithi055.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2023-12-19T19:03:36.567 INFO:teuthology.orchestra.run.smithi055.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2023-12-19T19:03:36.634 INFO:teuthology.orchestra.run.smithi055.stdout:(9/13): ceph-mgr-rook-17.2.7-131.ga88f3edf.el9. 76 kB/s | 47 kB 00:00 2023-12-19T19:03:36.758 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:04 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:36.903 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:36.904 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:36.904 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 2.9 M 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 12 M 2023-12-19T19:03:36.905 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:36.917 INFO:teuthology.orchestra.run.smithi055.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 526 kB/s | 197 kB 00:00 2023-12-19T19:03:36.976 INFO:teuthology.orchestra.run.smithi055.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2023-12-19T19:03:37.043 INFO:teuthology.orchestra.run.smithi055.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 718 kB/s | 90 kB 00:00 2023-12-19T19:03:37.084 INFO:teuthology.orchestra.run.smithi055.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2023-12-19T19:03:37.086 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:37.087 INFO:teuthology.orchestra.run.smithi055.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2023-12-19T19:03:37.087 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:37.103 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:37.103 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:37.218 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:37.220 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:37.770 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:37.947 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-19T19:03:38.004 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-19T19:03:38.059 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-19T19:03:38.096 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-19T19:03:38.138 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-19T19:03:38.233 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-19T19:03:38.343 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-19T19:03:38.397 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-19T19:03:39.157 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-19T19:03:39.210 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-19T19:03:39.244 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-19T19:03:39.290 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-19T19:03:39.314 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T19:03:39.390 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-17.2.7-131.ga88f3edf.el9.x86_64.rpm 1.2 MB/s | 2.9 MB 00:02 2023-12-19T19:03:39.392 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:39.393 INFO:teuthology.orchestra.run.smithi187.stdout:Total 1.2 MB/s | 2.9 MB 00:02 2023-12-19T19:03:39.393 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:39.402 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:39.404 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:39.479 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:39.481 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:39.897 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:39.958 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:39.982 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:39.982 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:03:39.982 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-19T19:03:39.982 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T19:03:39.982 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T19:03:39.983 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:40.156 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 13/13 2023-12-19T19:03:40.156 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 1/13 2023-12-19T19:03:40.156 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-19T19:03:40.157 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-rook-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-19T19:03:40.536 INFO:teuthology.orchestra.run.smithi055.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:40.537 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:40.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-mgr-cephadm 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:41.121 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:41.348 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:09 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:41.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2023-12-19T19:03:41.502 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:41.502 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:41.502 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm noarch 2:17.2.7-131.ga88f3edf.el9 ceph-noarch 121 k 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:======================================================================================= 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout:Install 6 Packages 2023-12-19T19:03:41.503 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:41.504 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 6.8 M 2023-12-19T19:03:41.504 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 33 M 2023-12-19T19:03:41.504 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:41.930 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:02:09 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:42.029 INFO:teuthology.orchestra.run.smithi055.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-19T19:03:42.075 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 162 k 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2023-12-19T19:03:42.076 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:42.077 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 162 k 2023-12-19T19:03:42.077 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 465 k 2023-12-19T19:03:42.077 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-12-19T19:03:42.312 INFO:teuthology.orchestra.run.smithi055.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2023-12-19T19:03:42.571 INFO:teuthology.orchestra.run.smithi055.stdout:(3/6): ceph-mgr-cephadm-17.2.7-131.ga88f3edf.el 173 kB/s | 121 kB 00:00 2023-12-19T19:03:42.743 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-17.2.7-131.ga88f3edf.el9.x86_64.rpm 243 kB/s | 162 kB 00:00 2023-12-19T19:03:42.744 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:42.744 INFO:teuthology.orchestra.run.smithi187.stdout:Total 243 kB/s | 162 kB 00:00 2023-12-19T19:03:42.744 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-12-19T19:03:42.753 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-12-19T19:03:42.753 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-12-19T19:03:42.763 INFO:teuthology.orchestra.run.smithi055.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2023-12-19T19:03:42.809 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-12-19T19:03:42.809 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-12-19T19:03:42.815 INFO:teuthology.orchestra.run.smithi055.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 697 kB/s | 548 kB 00:00 2023-12-19T19:03:42.960 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-12-19T19:03:43.005 INFO:teuthology.orchestra.run.smithi055.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2023-12-19T19:03:43.009 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:43.010 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.5 MB/s | 6.8 MB 00:01 2023-12-19T19:03:43.010 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:43.020 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:43.020 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:43.058 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:43.117 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:43.117 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:43.421 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:43.510 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:43.594 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-19T19:03:43.649 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-19T19:03:43.770 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:43.770 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:43.771 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-12-19T19:03:43.771 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:43.771 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:03:43.771 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-12-19T19:03:43.981 DEBUG:teuthology.parallel:result is None 2023-12-19T19:03:44.204 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-19T19:03:44.286 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-19T19:03:44.340 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-19T19:03:44.369 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T19:03:45.111 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 6/6 2023-12-19T19:03:45.112 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 1/6 2023-12-19T19:03:45.112 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-19T19:03:45.112 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-19T19:03:45.112 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-19T19:03:45.112 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-19T19:03:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-19T19:03:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:45.434 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-cephadm-2:17.2.7-131.ga88f3edf.el9.noarch 2023-12-19T19:03:45.434 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:45.435 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:45.663 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install ceph-fuse 2023-12-19T19:03:46.243 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:14 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repository Size 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 810 k 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout:Installing dependencies: 2023-12-19T19:03:46.397 INFO:teuthology.orchestra.run.smithi055.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:Install 2 Packages 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 889 k 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 2.6 M 2023-12-19T19:03:46.398 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:46.532 INFO:teuthology.orchestra.run.smithi055.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 592 kB/s | 79 kB 00:00 2023-12-19T19:03:47.557 INFO:teuthology.orchestra.run.smithi055.stdout:(2/2): ceph-fuse-17.2.7-131.ga88f3edf.el9.x86_6 700 kB/s | 810 kB 00:01 2023-12-19T19:03:47.558 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:47.558 INFO:teuthology.orchestra.run.smithi055.stdout:Total 766 kB/s | 889 kB 00:01 2023-12-19T19:03:47.559 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:47.585 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:47.585 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:47.670 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:47.670 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:47.871 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:47.998 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-19T19:03:48.101 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T19:03:48.770 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2/2 2023-12-19T19:03:48.770 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/2 2023-12-19T19:03:49.124 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-19T19:03:49.124 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:49.124 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:49.125 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-19T19:03:49.125 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:49.125 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:49.398 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados-devel 2023-12-19T19:03:49.986 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:17 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:50.138 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:50.138 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 126 k 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 126 k 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 448 k 2023-12-19T19:03:50.139 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:50.811 INFO:teuthology.orchestra.run.smithi055.stdout:librados-devel-17.2.7-131.ga88f3edf.el9.x86_64. 188 kB/s | 126 kB 00:00 2023-12-19T19:03:50.811 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:50.812 INFO:teuthology.orchestra.run.smithi055.stdout:Total 188 kB/s | 126 kB 00:00 2023-12-19T19:03:50.812 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:50.818 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:50.818 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:50.879 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:50.880 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:51.030 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:51.146 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:51.574 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:51.956 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:51.956 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:51.957 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:51.957 INFO:teuthology.orchestra.run.smithi055.stdout: librados-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:51.957 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:51.957 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:52.200 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs2 2023-12-19T19:03:52.786 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:20 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:52.885 INFO:teuthology.orchestra.run.smithi055.stdout:Package libcephfs2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:52.938 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:52.940 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:52.940 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:53.002 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install libcephfs-devel 2023-12-19T19:03:53.585 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:21 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:53.737 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 25 k 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:03:53.738 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:53.739 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 25 k 2023-12-19T19:03:53.739 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 110 k 2023-12-19T19:03:53.739 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:03:54.089 INFO:teuthology.orchestra.run.smithi055.stdout:libcephfs-devel-17.2.7-131.ga88f3edf.el9.x86_64 71 kB/s | 25 kB 00:00 2023-12-19T19:03:54.090 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:03:54.091 INFO:teuthology.orchestra.run.smithi055.stdout:Total 71 kB/s | 25 kB 00:00 2023-12-19T19:03:54.091 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:03:54.094 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:03:54.094 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:03:54.095 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:03:54.096 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:03:54.179 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:03:54.298 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:54.675 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:54.939 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:03:54.939 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:54.939 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:03:54.939 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-devel-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:03:54.940 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:03:54.940 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:55.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librados2 2023-12-19T19:03:55.723 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:23 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:55.823 INFO:teuthology.orchestra.run.smithi055.stdout:Package librados2-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:55.876 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:55.877 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:55.878 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:55.957 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install librbd1 2023-12-19T19:03:56.539 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:24 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:56.637 INFO:teuthology.orchestra.run.smithi055.stdout:Package librbd1-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:56.690 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:56.691 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:56.691 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:56.763 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rados 2023-12-19T19:03:57.345 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:25 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:57.443 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rados-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:57.496 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:57.497 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:57.497 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:57.565 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rgw 2023-12-19T19:03:58.151 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:26 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:58.247 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rgw-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:58.301 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:58.301 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:58.302 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:58.374 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-cephfs 2023-12-19T19:03:58.957 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:26 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:59.055 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-cephfs-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:59.108 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:59.109 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:59.109 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:59.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install python3-rbd 2023-12-19T19:03:59.763 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:27 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:03:59.861 INFO:teuthology.orchestra.run.smithi055.stdout:Package python3-rbd-2:17.2.7-131.ga88f3edf.el9.x86_64 is already installed. 2023-12-19T19:03:59.914 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:03:59.915 INFO:teuthology.orchestra.run.smithi055.stdout:Nothing to do. 2023-12-19T19:03:59.915 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:03:59.991 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-fuse 2023-12-19T19:04:00.569 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:28 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:04:00.720 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:04:00.722 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:00.722 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-12-19T19:04:00.722 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:00.722 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:04:00.722 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 81 k 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 81 k 2023-12-19T19:04:00.723 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 222 k 2023-12-19T19:04:00.724 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:04:01.026 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-fuse-17.2.7-131.ga88f3edf.el9.x86_64.rpm 267 kB/s | 81 kB 00:00 2023-12-19T19:04:01.026 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:04:01.027 INFO:teuthology.orchestra.run.smithi055.stdout:Total 266 kB/s | 81 kB 00:00 2023-12-19T19:04:01.027 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:04:01.034 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:04:01.034 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:04:01.092 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:04:01.093 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:04:01.238 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:04:01.486 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:01.860 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-fuse-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:02.165 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:04:02.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-mirror 2023-12-19T19:04:02.968 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:30 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:04:03.122 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout: Package Arch Version Repo Size 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 2.9 M 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:04:03.123 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:03.124 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 2.9 M 2023-12-19T19:04:03.124 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 12 M 2023-12-19T19:04:03.124 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:04:03.821 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-mirror-17.2.7-131.ga88f3edf.el9.x86_64.rpm 4.2 MB/s | 2.9 MB 00:00 2023-12-19T19:04:03.822 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:04:03.822 INFO:teuthology.orchestra.run.smithi055.stdout:Total 4.2 MB/s | 2.9 MB 00:00 2023-12-19T19:04:03.822 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:04:03.832 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:04:03.832 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:04:03.907 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:04:03.907 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:04:04.335 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:04:04.397 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T19:04:04.424 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-mirror-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:05.622 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:04:05.867 DEBUG:teuthology.orchestra.run.smithi055:> sudo yum -y install rbd-nbd 2023-12-19T19:04:06.449 INFO:teuthology.orchestra.run.smithi055.stdout:Last metadata expiration check: 0:02:34 ago on Tue 19 Dec 2023 07:01:32 PM UTC. 2023-12-19T19:04:06.602 INFO:teuthology.orchestra.run.smithi055.stdout:Dependencies resolved. 2023-12-19T19:04:06.602 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:06.602 INFO:teuthology.orchestra.run.smithi055.stdout: Package Architecture Version Repository Size 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Installing: 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd x86_64 2:17.2.7-131.ga88f3edf.el9 ceph 162 k 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction Summary 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:================================================================================ 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Install 1 Package 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Total download size: 162 k 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Installed size: 465 k 2023-12-19T19:04:06.603 INFO:teuthology.orchestra.run.smithi055.stdout:Downloading Packages: 2023-12-19T19:04:06.958 INFO:teuthology.orchestra.run.smithi055.stdout:rbd-nbd-17.2.7-131.ga88f3edf.el9.x86_64.rpm 458 kB/s | 162 kB 00:00 2023-12-19T19:04:06.958 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:04:06.959 INFO:teuthology.orchestra.run.smithi055.stdout:Total 456 kB/s | 162 kB 00:00 2023-12-19T19:04:06.959 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction check 2023-12-19T19:04:06.967 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction check succeeded. 2023-12-19T19:04:06.967 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction test 2023-12-19T19:04:07.027 INFO:teuthology.orchestra.run.smithi055.stdout:Transaction test succeeded. 2023-12-19T19:04:07.027 INFO:teuthology.orchestra.run.smithi055.stdout:Running transaction 2023-12-19T19:04:07.180 INFO:teuthology.orchestra.run.smithi055.stdout: Preparing : 1/1 2023-12-19T19:04:07.283 INFO:teuthology.orchestra.run.smithi055.stdout: Installing : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:07.866 INFO:teuthology.orchestra.run.smithi055.stdout: Running scriptlet: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:08.180 INFO:teuthology.orchestra.run.smithi055.stdout: Verifying : rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 1/1 2023-12-19T19:04:08.180 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:08.180 INFO:teuthology.orchestra.run.smithi055.stdout:Installed: 2023-12-19T19:04:08.180 INFO:teuthology.orchestra.run.smithi055.stdout: rbd-nbd-2:17.2.7-131.ga88f3edf.el9.x86_64 2023-12-19T19:04:08.180 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:08.181 INFO:teuthology.orchestra.run.smithi055.stdout:Complete! 2023-12-19T19:04:08.404 DEBUG:teuthology.parallel:result is None 2023-12-19T19:04:08.405 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:04:08.405 INFO:teuthology.packaging:ref: None 2023-12-19T19:04:08.405 INFO:teuthology.packaging:tag: None 2023-12-19T19:04:08.405 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:04:08.405 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:04:08.406 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T19:04:08.619 DEBUG:teuthology.orchestra.run.smithi055:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-19T19:04:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:17.2.7-131.ga88f3edf.el9 2023-12-19T19:04:08.641 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-131.ga88f3edf.el9 2023-12-19T19:04:08.641 INFO:teuthology.task.install:The correct ceph version 17.2.7-131.ga88f3edf is installed. 2023-12-19T19:04:08.643 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T19:04:08.643 INFO:teuthology.packaging:ref: None 2023-12-19T19:04:08.643 INFO:teuthology.packaging:tag: None 2023-12-19T19:04:08.643 INFO:teuthology.packaging:branch: quincy 2023-12-19T19:04:08.643 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:04:08.643 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-19T19:04:08.945 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-19T19:04:08.970 INFO:teuthology.orchestra.run.smithi187.stdout:17.2.7-131.ga88f3edf.el9 2023-12-19T19:04:08.970 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-131.ga88f3edf.el9 2023-12-19T19:04:08.971 INFO:teuthology.task.install:The correct ceph version 17.2.7-131.ga88f3edf is installed. 2023-12-19T19:04:08.972 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-19T19:04:08.972 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:04:08.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T19:04:09.005 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:04:09.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T19:04:09.043 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-19T19:04:09.043 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:04:09.044 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T19:04:09.078 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T19:04:09.153 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:04:09.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T19:04:09.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T19:04:09.253 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-19T19:04:09.254 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:04:09.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T19:04:09.286 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T19:04:09.358 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:04:09.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T19:04:09.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T19:04:09.455 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-19T19:04:09.456 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:04:09.456 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T19:04:09.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T19:04:09.561 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:04:09.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T19:04:09.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T19:04:09.655 INFO:teuthology.run_tasks:Running task print... 2023-12-19T19:04:09.666 INFO:teuthology.task.print:**** done install task... 2023-12-19T19:04:09.666 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-19T19:04:09.751 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-19T19:04:09.751 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2023-12-19T19:04:09.752 INFO:tasks.cephadm:Cluster fsid is 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:04:09.752 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-19T19:04:09.752 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-19T19:04:09.752 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi055': '172.21.15.55', 'mon.smithi187': '172.21.15.187'} 2023-12-19T19:04:09.752 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-19T19:04:09.752 DEBUG:teuthology.orchestra.run.smithi055:> sudo hostname $(hostname -s) 2023-12-19T19:04:09.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2023-12-19T19:04:09.813 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2023-12-19T19:04:09.813 DEBUG:teuthology.orchestra.run.smithi055:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.075 INFO:teuthology.orchestra.run.smithi055.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 19 19:04 /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.075 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.240 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 19 19:04 /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.240 DEBUG:teuthology.orchestra.run.smithi055:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.258 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-19T19:04:10.284 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2023-12-19T19:04:10.285 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-19T19:04:10.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-19T19:04:11.191 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T19:04:11.405 INFO:teuthology.orchestra.run.smithi055.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T19:04:46.075 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2023-12-19T19:04:46.075 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)", 2023-12-19T19:04:46.076 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e", 2023-12-19T19:04:46.076 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2023-12-19T19:04:46.076 INFO:teuthology.orchestra.run.smithi187.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd" 2023-12-19T19:04:46.076 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2023-12-19T19:04:46.076 INFO:teuthology.orchestra.run.smithi187.stdout:} 2023-12-19T19:04:49.145 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph_version": "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)", 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout: "image_id": "f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e", 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout: "repo_digests": [ 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd" 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout: ] 2023-12-19T19:04:49.146 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-19T19:04:49.169 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph 2023-12-19T19:04:49.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2023-12-19T19:04:49.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /etc/ceph 2023-12-19T19:04:49.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2023-12-19T19:04:49.310 INFO:tasks.cephadm:Writing seed config... 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-19T19:04:49.312 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-19T19:04:49.313 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-19T19:04:49.314 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-19T19:04:49.315 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-19T19:04:49.315 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-19T19:04:49.315 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-19T19:04:49.315 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-19T19:04:49.316 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:04:49.316 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-19T19:04:49.336 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 63de9ba6-9ea1-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-19T19:04:49.337 DEBUG:teuthology.orchestra.run.smithi055:mon.smithi055> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service 2023-12-19T19:04:49.379 INFO:tasks.cephadm:Bootstrapping... 2023-12-19T19:04:49.380 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.55 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-19T19:04:49.596 INFO:teuthology.orchestra.run.smithi055.stdout:-------------------------------------------------------------------------------- 2023-12-19T19:04:49.597 INFO:teuthology.orchestra.run.smithi055.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '63de9ba6-9ea1-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.55', '--skip-admin-label'] 2023-12-19T19:04:49.621 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.8.1 2023-12-19T19:04:49.621 INFO:teuthology.orchestra.run.smithi055.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-12-19T19:04:49.621 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying podman|docker is present... 2023-12-19T19:04:49.644 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.8.1 2023-12-19T19:04:49.645 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying lvm2 is present... 2023-12-19T19:04:49.645 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying time synchronization is in place... 2023-12-19T19:04:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-19T19:04:49.652 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-19T19:04:49.659 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-19T19:04:49.659 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:04:49.666 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2023-12-19T19:04:49.672 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2023-12-19T19:04:49.672 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2023-12-19T19:04:49.673 INFO:teuthology.orchestra.run.smithi055.stdout:Repeating the final host check... 2023-12-19T19:04:49.696 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout 4.8.1 2023-12-19T19:04:49.696 INFO:teuthology.orchestra.run.smithi055.stdout:podman (/bin/podman) version 4.8.1 is present 2023-12-19T19:04:49.696 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl is present 2023-12-19T19:04:49.696 INFO:teuthology.orchestra.run.smithi055.stdout:lvcreate is present 2023-12-19T19:04:49.702 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-19T19:04:49.703 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-19T19:04:49.709 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-19T19:04:49.709 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:04:49.715 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout enabled 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout active 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:Unit chronyd.service is enabled and running 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:Host looks OK 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:Cluster fsid: 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:Acquiring lock 139820650541120 on /run/cephadm/63de9ba6-9ea1-11ee-95a5-87774f69a715.lock 2023-12-19T19:04:49.722 INFO:teuthology.orchestra.run.smithi055.stdout:Lock 139820650541120 acquired on /run/cephadm/63de9ba6-9ea1-11ee-95a5-87774f69a715.lock 2023-12-19T19:04:49.723 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 3300 ... 2023-12-19T19:04:49.723 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying IP 172.21.15.55 port 6789 ... 2023-12-19T19:04:49.723 INFO:teuthology.orchestra.run.smithi055.stdout:Base mon IP(s) is [172.21.15.55:3300, 172.21.15.55:6789], mon addrv is [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2023-12-19T19:04:49.727 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.55 metric 100 2023-12-19T19:04:49.727 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.55 metric 100 2023-12-19T19:04:49.730 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-12-19T19:04:49.730 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2023-12-19T19:04:49.730 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2023-12-19T19:04:49.734 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-19T19:04:49.734 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-12-19T19:04:49.734 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-19T19:04:49.734 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:Mon IP `172.21.15.55` is in CIDR network `172.21.0.0/20` 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-12-19T19:04:49.735 INFO:teuthology.orchestra.run.smithi055.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-19T19:04:49.736 INFO:teuthology.orchestra.run.smithi055.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T19:04:50.171 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stdout f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e 2023-12-19T19:04:50.172 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-19T19:04:50.172 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Getting image source signatures 2023-12-19T19:04:50.172 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:7be6900e5445e78fc03bcf48e8188d376353468c6b158a72fddc62646d503826 2023-12-19T19:04:50.172 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2023-12-19T19:04:50.172 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Copying config sha256:f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e 2023-12-19T19:04:50.173 INFO:teuthology.orchestra.run.smithi055.stdout:/bin/podman: stderr Writing manifest to image destination 2023-12-19T19:04:51.060 INFO:teuthology.orchestra.run.smithi055.stdout:ceph: stdout ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable) 2023-12-19T19:04:51.060 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph version: ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable) 2023-12-19T19:04:51.060 INFO:teuthology.orchestra.run.smithi055.stdout:Extracting ceph user uid/gid from container image... 2023-12-19T19:04:51.917 INFO:teuthology.orchestra.run.smithi055.stdout:stat: stdout 167 167 2023-12-19T19:04:51.917 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial keys... 2023-12-19T19:04:52.790 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQBU6YFl8ubIDhAASXWVDSntC/2RaXDBalYBIw== 2023-12-19T19:04:53.589 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQBV6YFlw71IBhAAkXNqEYs+sSjKJ6swBtw2aw== 2023-12-19T19:04:54.604 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph-authtool: stdout AQBV6YFldZhoNxAA1CkbrhDZVhAQdXroSvzS1A== 2023-12-19T19:04:54.605 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial monmap... 2023-12-19T19:04:55.645 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-19T19:04:55.645 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-12-19T19:04:55.645 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:04:55.645 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-19T19:04:55.645 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool for smithi055 [v2:172.21.15.55:3300,v1:172.21.15.55:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-19T19:04:55.646 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = octopus 2023-12-19T19:04:55.646 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: set fsid to 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:04:55.646 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-19T19:04:55.646 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:04:55.646 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mon... 2023-12-19T19:04:56.510 INFO:teuthology.orchestra.run.smithi055.stdout:create mon.smithi055 on 2023-12-19T19:04:56.694 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2023-12-19T19:04:56.867 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-19T19:04:57.023 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target. 2023-12-19T19:04:57.023 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target. 2023-12-19T19:04:57.207 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055 2023-12-19T19:04:57.207 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service: Unit ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service not loaded. 2023-12-19T19:04:57.375 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target.wants/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service → /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@.service. 2023-12-19T19:04:57.875 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T19:04:57.875 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon to start... 2023-12-19T19:04:57.883 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mon... 2023-12-19T19:04:58.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:04:58 smithi055 ceph-mon[80602]: from='client.? 172.21.15.55:0/699000625' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-19T19:04:58.944 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout cluster: 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout id: 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout services: 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi055 (age 0.539567s) 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout data: 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-12-19T19:04:58.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout pgs: 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.287+0000 7fe7b89d1700 1 Processor -- start 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.287+0000 7fe7b89d1700 1 -- start start 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b89d1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b89d1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4077770 con 0x7fe7b4078e20 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38898/0 (socket says 172.21.15.55:38898) 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b259c700 1 -- 172.21.15.55:0/484157295 learned_addr learned my addr 172.21.15.55:0/484157295 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.288+0000 7fe7b259c700 1 -- 172.21.15.55:0/484157295 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40778b0 con 0x7fe7b4078e20 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b259c700 1 --2- 172.21.15.55:0/484157295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40098e0 tx=0x7fe7a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1026584a4bd58d server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b159a700 1 -- 172.21.15.55:0/484157295 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a400b860 con 0x7fe7b4078e20 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b159a700 1 -- 172.21.15.55:0/484157295 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a400b9c0 con 0x7fe7b4078e20 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b159a700 1 -- 172.21.15.55:0/484157295 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a400bc90 con 0x7fe7b4078e20 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b89d1700 1 -- 172.21.15.55:0/484157295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 msgr2=0x7fe7b4077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:04:58.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b89d1700 1 --2- 172.21.15.55:0/484157295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40098e0 tx=0x7fe7a4009250 comp rx=0 tx=0).stop 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b89d1700 1 -- 172.21.15.55:0/484157295 shutdown_connections 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b89d1700 1 --2- 172.21.15.55:0/484157295 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4077230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.289+0000 7fe7b89d1700 1 -- 172.21.15.55:0/484157295 >> 172.21.15.55:0/484157295 conn(0x7fe7b40fda40 msgr2=0x7fe7b40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 -- 172.21.15.55:0/484157295 shutdown_connections 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 -- 172.21.15.55:0/484157295 wait complete. 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 Processor -- start 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 -- start start 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b89d1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41130f0 con 0x7fe7b4078e20 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38910/0 (socket says 172.21.15.55:38910) 2023-12-19T19:04:58.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.290+0000 7fe7b259c700 1 -- 172.21.15.55:0/699000625 learned_addr learned my addr 172.21.15.55:0/699000625 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7b259c700 1 -- 172.21.15.55:0/699000625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a40094f0 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7b259c700 1 --2- 172.21.15.55:0/699000625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009290 tx=0x7fe7a4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a401c5c0 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a401c740 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b41132f0 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a401ca50 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b4113700 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe7a401a450 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.291+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe7a402e790 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.292+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4077bc0 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.294+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe7a401ce30 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.369+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe7b410bea0 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7a2ffd700 1 -- 172.21.15.55:0/699000625 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe7a401a890 con 0x7fe7b4078e20 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 msgr2=0x7fe7b4112bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:04:58.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7b89d1700 1 --2- 172.21.15.55:0/699000625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009290 tx=0x7fe7a4016000 comp rx=0 tx=0).stop 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 shutdown_connections 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7b89d1700 1 --2- 172.21.15.55:0/699000625 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe7b4078e20 0x7fe7b4112bb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.370+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 >> 172.21.15.55:0/699000625 conn(0x7fe7b40fda40 msgr2=0x7fe7b4107040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.371+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 shutdown_connections 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:58.371+0000 7fe7b89d1700 1 -- 172.21.15.55:0/699000625 wait complete. 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:mon is available 2023-12-19T19:04:58.949 INFO:teuthology.orchestra.run.smithi055.stdout:Assimilating anything we can from ceph.conf... 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.55:3300,v1:172.21.15.55:6789] 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2023-12-19T19:05:00.019 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f030a5ba700 1 Processor -- start 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f030a5ba700 1 -- start start 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f030a5ba700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f030a5ba700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0304077770 con 0x7f0304078e20 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f0303fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f0303fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38922/0 (socket says 172.21.15.55:38922) 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.367+0000 7f0303fff700 1 -- 172.21.15.55:0/3696166652 learned_addr learned my addr 172.21.15.55:0/3696166652 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f0303fff700 1 -- 172.21.15.55:0/3696166652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040778b0 con 0x7f0304078e20 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f0303fff700 1 --2- 172.21.15.55:0/3696166652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f02f40098e0 tx=0x7f02f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a753132fe171b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f0302ffd700 1 -- 172.21.15.55:0/3696166652 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f400b860 con 0x7f0304078e20 2023-12-19T19:05:00.020 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f0302ffd700 1 -- 172.21.15.55:0/3696166652 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f400b9c0 con 0x7f0304078e20 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f0302ffd700 1 -- 172.21.15.55:0/3696166652 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f400bc90 con 0x7f0304078e20 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f030a5ba700 1 -- 172.21.15.55:0/3696166652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 msgr2=0x7f0304077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f030a5ba700 1 --2- 172.21.15.55:0/3696166652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f02f40098e0 tx=0x7f02f4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f030a5ba700 1 -- 172.21.15.55:0/3696166652 shutdown_connections 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f030a5ba700 1 --2- 172.21.15.55:0/3696166652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304077230 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.368+0000 7f030a5ba700 1 -- 172.21.15.55:0/3696166652 >> 172.21.15.55:0/3696166652 conn(0x7f03040fda40 msgr2=0x7f03040ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.369+0000 7f030a5ba700 1 -- 172.21.15.55:0/3696166652 shutdown_connections 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.369+0000 7f030a5ba700 1 -- 172.21.15.55:0/3696166652 wait complete. 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.369+0000 7f030a5ba700 1 Processor -- start 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.369+0000 7f030a5ba700 1 -- start start 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f030a5ba700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f030a5ba700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03041130f0 con 0x7f0304078e20 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0303fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:00.021 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0303fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38926/0 (socket says 172.21.15.55:38926) 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0303fff700 1 -- 172.21.15.55:0/1464068464 learned_addr learned my addr 172.21.15.55:0/1464068464 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0303fff700 1 -- 172.21.15.55:0/1464068464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40094f0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0303fff700 1 --2- 172.21.15.55:0/1464068464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009290 tx=0x7f02f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f401c5c0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f401c740 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03041132f0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02f401ca50 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.370+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0304113700 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.371+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f02f401a450 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.371+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f02f402e790 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.372+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304077bc0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.374+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f02f401ce30 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.444+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f030410bfc0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.452+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f02f401b3e0 con 0x7f0304078e20 2023-12-19T19:05:00.022 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.452+0000 7f0300ff9700 1 -- 172.21.15.55:0/1464068464 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f02f402e950 con 0x7f0304078e20 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 msgr2=0x7f0304112bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 --2- 172.21.15.55:0/1464068464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009290 tx=0x7f02f4016000 comp rx=0 tx=0).stop 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 shutdown_connections 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 --2- 172.21.15.55:0/1464068464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0304078e20 0x7f0304112bb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 >> 172.21.15.55:0/1464068464 conn(0x7f03040fda40 msgr2=0x7f0304107040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 shutdown_connections 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:04:59.453+0000 7f030a5ba700 1 -- 172.21.15.55:0/1464068464 wait complete. 2023-12-19T19:05:00.023 INFO:teuthology.orchestra.run.smithi055.stdout:Generating new minimal ceph.conf... 2023-12-19T19:05:00.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:04:59 smithi055 ceph-mon[80602]: from='client.? 172.21.15.55:0/1464068464' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-19T19:05:00.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:04:59 smithi055 ceph-mon[80602]: from='client.? 172.21.15.55:0/1464068464' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-19T19:05:01.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:00 smithi055 ceph-mon[80602]: from='client.? 172.21.15.55:0/1552476917' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538f094700 1 Processor -- start 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538f094700 1 -- start start 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538f094700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538f094700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388104310 con 0x7f53881039c0 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538ce30700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:01.074 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538ce30700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38938/0 (socket says 172.21.15.55:38938) 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.415+0000 7f538ce30700 1 -- 172.21.15.55:0/1865501155 learned_addr learned my addr 172.21.15.55:0/1865501155 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f538ce30700 1 -- 172.21.15.55:0/1865501155 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388101fb0 con 0x7f53881039c0 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f538ce30700 1 --2- 172.21.15.55:0/1865501155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f537c0098e0 tx=0x7f537c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36547e32f1014ec5 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f53877fe700 1 -- 172.21.15.55:0/1865501155 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f537c00b860 con 0x7f53881039c0 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f53877fe700 1 -- 172.21.15.55:0/1865501155 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f537c00be30 con 0x7f53881039c0 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f53877fe700 1 -- 172.21.15.55:0/1865501155 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f537c010380 con 0x7f53881039c0 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f538f094700 1 -- 172.21.15.55:0/1865501155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 msgr2=0x7f5388103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.416+0000 7f538f094700 1 --2- 172.21.15.55:0/1865501155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f537c0098e0 tx=0x7f537c009250 comp rx=0 tx=0).stop 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 -- 172.21.15.55:0/1865501155 shutdown_connections 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 --2- 172.21.15.55:0/1865501155 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f5388103dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 -- 172.21.15.55:0/1865501155 >> 172.21.15.55:0/1865501155 conn(0x7f53880fda40 msgr2=0x7f53880ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 -- 172.21.15.55:0/1865501155 shutdown_connections 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 -- 172.21.15.55:0/1865501155 wait complete. 2023-12-19T19:05:01.075 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 Processor -- start 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 -- start start 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.417+0000 7f538f094700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538f094700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5388111630 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538ce30700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538ce30700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38940/0 (socket says 172.21.15.55:38940) 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538ce30700 1 -- 172.21.15.55:0/1552476917 learned_addr learned my addr 172.21.15.55:0/1552476917 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538ce30700 1 -- 172.21.15.55:0/1552476917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c0094f0 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538ce30700 1 --2- 172.21.15.55:0/1552476917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f537c000c00 tx=0x7f537c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f537c010a80 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f537c01a430 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5388111830 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f537c0245b0 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5388111c40 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.418+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f537c024aa0 con 0x7f53881039c0 2023-12-19T19:05:01.076 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.419+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f537c02eae0 con 0x7f53881039c0 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.419+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5388102d60 con 0x7f53881039c0 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.422+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f537c029070 con 0x7f53881039c0 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.492+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f538810adb0 con 0x7f53881039c0 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.493+0000 7f53857fa700 1 -- 172.21.15.55:0/1552476917 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f537c010be0 con 0x7f53881039c0 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.493+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 msgr2=0x7f53881110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.493+0000 7f538f094700 1 --2- 172.21.15.55:0/1552476917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f537c000c00 tx=0x7f537c009250 comp rx=0 tx=0).stop 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.494+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 shutdown_connections 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.494+0000 7f538f094700 1 --2- 172.21.15.55:0/1552476917 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53881039c0 0x7f53881110f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.494+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 >> 172.21.15.55:0/1552476917 conn(0x7f53880fda40 msgr2=0x7f53880ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.494+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 shutdown_connections 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:00.494+0000 7f538f094700 1 -- 172.21.15.55:0/1552476917 wait complete. 2023-12-19T19:05:01.077 INFO:teuthology.orchestra.run.smithi055.stdout:Restarting the monitor... 2023-12-19T19:05:01.478 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:05:01.478 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80596]: 2023-12-19T19:05:01.302+0000 7fcade286700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:05:01.478 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80596]: 2023-12-19T19:05:01.302+0000 7fcade286700 -1 mon.smithi055@0(leader) e1 *** Got Signal Terminated *** 2023-12-19T19:05:01.778 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 podman[80849]: 2023-12-19 19:05:01.47699378 +0000 UTC m=+0.249904464 container died 3576ed31d89a44d80389de58762796633ec1dcd73f7e7a9e920a149b26cf9006 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2023-12-19T19:05:01.779 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 podman[80849]: 2023-12-19 19:05:01.777534691 +0000 UTC m=+0.550445366 container cleanup 3576ed31d89a44d80389de58762796633ec1dcd73f7e7a9e920a149b26cf9006 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0) 2023-12-19T19:05:02.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 bash[80849]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055 2023-12-19T19:05:02.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:01 smithi055 podman[80863]: 2023-12-19 19:05:01.978723493 +0000 UTC m=+0.501414328 container remove 3576ed31d89a44d80389de58762796633ec1dcd73f7e7a9e920a149b26cf9006 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2023-12-19T19:05:02.319 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service: Deactivated successfully. 2023-12-19T19:05:02.319 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 systemd[1]: Stopped Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:05:02.320 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 systemd[1]: Starting Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:05:02.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 podman[80959]: 2023-12-19 19:05:02.220990139 +0000 UTC m=+0.019216664 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph:quincy 2023-12-19T19:05:02.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 podman[80959]: 2023-12-19 19:05:02.333341232 +0000 UTC m=+0.131567733 container create 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux ) 2023-12-19T19:05:02.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 podman[80959]: 2023-12-19 19:05:02.491000159 +0000 UTC m=+0.289226666 container init 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-a88f3ed, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2023-12-19T19:05:02.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 podman[80959]: 2023-12-19 19:05:02.493856907 +0000 UTC m=+0.292083408 container start 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, RELEASE=quincy-a88f3ed, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, GIT_BRANCH=HEAD) 2023-12-19T19:05:02.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: set uid:gid to 167:167 (ceph:ceph) 2023-12-19T19:05:02.517 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable), process ceph-mon, pid 2 2023-12-19T19:05:02.517 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: pidfile_write: ignore empty --pid-file 2023-12-19T19:05:02.554 INFO:teuthology.orchestra.run.smithi055.stdout:Setting mon public_network to 172.21.0.0/20 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: load: jerasure load: lrc 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: RocksDB version: 6.15.5 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Compile date Dec 19 2023 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: DB SUMMARY 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: DB Session ID: 6CHXG14W7R4QNJKLNDAI 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: CURRENT file: CURRENT 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: IDENTITY file: IDENTITY 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-12-19T19:05:02.595 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi055/store.db dir, Total Num: 1, files: 000008.sst 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi055/store.db: 000010.log size: 96760 ; 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.error_if_exists: 0 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.create_if_missing: 0 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.paranoid_checks: 1 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.env: 0x55db86be3a80 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.fs: Posix File System 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.info_log: 0x55db884ad000 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_file_opening_threads: 16 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.statistics: (nil) 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.use_fsync: 0 2023-12-19T19:05:02.596 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_log_file_size: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.log_file_time_to_roll: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.keep_log_file_num: 1000 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.recycle_log_file_num: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_fallocate: 1 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_mmap_reads: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_mmap_writes: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.use_direct_reads: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.create_missing_column_families: 0 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.db_log_dir: 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi055/store.db 2023-12-19T19:05:02.597 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.table_cache_numshardbits: 6 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.WAL_ttl_seconds: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.WAL_size_limit_MB: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.is_fd_close_on_exec: 1 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.advise_random_on_open: 1 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.db_write_buffer_size: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.write_buffer_manager: 0x55db892229f0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.use_adaptive_mutex: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.rate_limiter: (nil) 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-19T19:05:02.598 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.wal_recovery_mode: 2 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.enable_thread_tracking: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.enable_pipelined_write: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.unordered_write: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.row_cache: None 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.wal_filter: None 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_ingest_behind: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.preserve_deletes: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.two_write_queues: 0 2023-12-19T19:05:02.599 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.manual_wal_flush: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.atomic_flush: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.persist_stats_to_disk: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.write_dbid_to_manifest: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.log_readahead_size: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.best_efforts_recovery: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.allow_data_in_errors: 0 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.db_host_id: __hostname__ 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_background_jobs: 2 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_background_compactions: -1 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_subcompactions: 1 2023-12-19T19:05:02.600 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.delayed_write_rate : 16777216 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_total_wal_size: 0 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.stats_dump_period_sec: 600 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.stats_persist_period_sec: 600 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_open_files: -1 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bytes_per_sync: 0 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.wal_bytes_per_sync: 0 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.strict_bytes_per_sync: 0 2023-12-19T19:05:02.602 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_readahead_size: 0 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_background_flushes: -1 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Compression algorithms supported: 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kZSTD supported: 0 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kXpressCompression supported: 0 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kLZ4HCCompression supported: 1 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kZlibCompression supported: 1 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kSnappyCompression supported: 1 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kLZ4Compression supported: 1 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: kBZip2Compression supported: 0 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Fast CRC32 supported: Supported on x86 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000009 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-19T19:05:02.603 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.merge_operator: 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_filter: None 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_filter_factory: None 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.sst_partitioner_factory: None 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.memtable_factory: SkipListFactory 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.table_factory: BlockBasedTable 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55db88502f90) 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: cache_index_and_filter_blocks: 1 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: pin_top_level_index_and_filter: 1 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_type: 0 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: data_block_index_type: 0 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_shortening: 1 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: data_block_hash_table_util_ratio: 0.750000 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: hash_index_allow_collision: 1 2023-12-19T19:05:02.604 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: checksum: 1 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: no_block_cache: 0 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache: 0x55db88535090 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_name: BinnedLRUCache 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_options: 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: capacity : 536870912 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: num_shard_bits : 4 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: strict_capacity_limit : 0 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: high_pri_pool_ratio: 0.000 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_cache_compressed: (nil) 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: persistent_cache: (nil) 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_size: 4096 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_size_deviation: 10 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_restart_interval: 16 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: index_block_restart_interval: 1 2023-12-19T19:05:02.605 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: metadata_block_size: 4096 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: partition_filters: 0 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: use_delta_encoding: 1 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: whole_key_filtering: 1 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: verify_compression: 0 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: read_amp_bytes_per_bit: 0 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: format_version: 4 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: enable_index_compression: 1 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: block_align: 0 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.write_buffer_size: 33554432 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_write_buffer_number: 2 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression: NoCompression 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression: Disabled 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.prefix_extractor: nullptr 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-19T19:05:02.606 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.num_levels: 7 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.window_bits: -14 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.level: 32767 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.strategy: 0 2023-12-19T19:05:02.607 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compression_opts.enabled: false 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.target_file_size_base: 67108864 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.target_file_size_multiplier: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-19T19:05:02.608 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.arena_block_size: 4194304 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.disable_auto_compactions: 0 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-19T19:05:02.609 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.table_properties_collectors: 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.inplace_update_support: 0 2023-12-19T19:05:02.610 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.inplace_update_num_locks: 10000 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.memtable_huge_page_size: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.bloom_locality: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.max_successive_merges: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.optimize_filters_for_hits: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.paranoid_file_checks: 0 2023-12-19T19:05:02.611 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.force_consistency_checks: 1 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.report_bg_io_stats: 0 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.ttl: 2592000 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.periodic_compaction_seconds: 0 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.enable_blob_files: false 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.min_blob_size: 0 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.blob_file_size: 268435456 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.blob_compression_type: NoCompression 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.enable_blob_garbage_collection: false 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi055/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703012702520546, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-12-19T19:05:02.612 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703012702521363, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92129, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90224, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82963, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703012702, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2339b3c5-0f44-4448-a643-9291cb07dc7a", "db_session_id": "6CHXG14W7R4QNJKLNDAI"}} 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: EVENT_LOG_v1 {"time_micros": 1703012702521919, "job": 1, "event": "recovery_finished"} 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi055/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55db884c9180 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: DB pointer 0x55db8859e000 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** DB Stats ** 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-19T19:05:02.613 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T19:05:02.614 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval compaction: 0.00 GB write, 23.47 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T19:05:02.615 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: L0 2/0 91.77 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Sum 2/0 91.77 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** Compaction Stats [default] ** 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.2 0.00 0.00 1 0.001 0 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Cumulative compaction: 0.00 GB write, 23.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-19T19:05:02.616 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: starting mon.smithi055 rank 0 at public addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] at bind addrs [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi055 fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???) e1 preinit fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).mds e1 new map 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).mds e1 print_map 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: e1 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: legacy client fscid: -1 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout: No filesystems configured 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T19:05:02.617 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 bash[80959]: 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 systemd[1]: Started Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mon.smithi055 is new leader, mons smithi055 in quorum (ranks 0) 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: monmap e1: 1 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: fsmap 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: osdmap e1: 0 total, 0 up, 0 in 2023-12-19T19:05:02.618 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:02 smithi055 ceph-mon[80978]: mgrmap e1: no daemons active 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f7501f9d700 1 Processor -- start 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f7501f9d700 1 -- start start 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f7501f9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f7501f9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc103230 con 0x7f74fc1028e0 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f74fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f74fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38954/0 (socket says 172.21.15.55:38954) 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.013+0000 7f74fb7fe700 1 -- 172.21.15.55:0/4210864430 learned_addr learned my addr 172.21.15.55:0/4210864430 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f74fb7fe700 1 -- 172.21.15.55:0/4210864430 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc103370 con 0x7f74fc1028e0 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f74fb7fe700 1 --2- 172.21.15.55:0/4210864430 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85d8cf8ca314758 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f74fa7fc700 1 -- 172.21.15.55:0/4210864430 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec00b8f0 con 0x7f74fc1028e0 2023-12-19T19:05:03.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f74fa7fc700 1 -- 172.21.15.55:0/4210864430 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f74ec011420 con 0x7f74fc1028e0 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f74fa7fc700 1 -- 172.21.15.55:0/4210864430 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec010540 con 0x7f74fc1028e0 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f7501f9d700 1 -- 172.21.15.55:0/4210864430 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 msgr2=0x7f74fc102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.014+0000 7f7501f9d700 1 --2- 172.21.15.55:0/4210864430 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0098e0 tx=0x7f74ec009250 comp rx=0 tx=0).stop 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- 172.21.15.55:0/4210864430 shutdown_connections 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 --2- 172.21.15.55:0/4210864430 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc102cf0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- 172.21.15.55:0/4210864430 >> 172.21.15.55:0/4210864430 conn(0x7f74fc0fc0b0 msgr2=0x7f74fc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- 172.21.15.55:0/4210864430 shutdown_connections 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- 172.21.15.55:0/4210864430 wait complete. 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 Processor -- start 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- start start 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.015+0000 7f7501f9d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc113940 con 0x7f74fc1028e0 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74fb7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38968/0 (socket says 172.21.15.55:38968) 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74fb7fe700 1 -- 172.21.15.55:0/654794312 learned_addr learned my addr 172.21.15.55:0/654794312 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:03.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74fb7fe700 1 -- 172.21.15.55:0/654794312 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec0094f0 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74fb7fe700 1 --2- 172.21.15.55:0/654794312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74ec011d20 tx=0x7f74ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec00be80 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f74ec010e80 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc113b40 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74ec01ae80 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc113f50 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f74ec024740 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.016+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f74ec02ea40 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.017+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc04bb80 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.019+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f74ec01a430 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.090+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f74fc10cd90 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.097+0000 7f74e3fff700 1 -- 172.21.15.55:0/654794312 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f74ec01a630 con 0x7f74fc1028e0 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 msgr2=0x7f74fc113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:03.643 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 --2- 172.21.15.55:0/654794312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f74ec011d20 tx=0x7f74ec009250 comp rx=0 tx=0).stop 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 shutdown_connections 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 --2- 172.21.15.55:0/654794312 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f74fc1028e0 0x7f74fc113400 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 >> 172.21.15.55:0/654794312 conn(0x7f74fc0fc0b0 msgr2=0x7f74fc0fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.098+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 shutdown_connections 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:03.099+0000 7f7501f9d700 1 -- 172.21.15.55:0/654794312 wait complete. 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote config to /etc/ceph/ceph.conf 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:Creating mgr... 2023-12-19T19:05:03.644 INFO:teuthology.orchestra.run.smithi055.stdout:Verifying port 9283 ... 2023-12-19T19:05:03.820 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mgr.smithi055.dwclvp 2023-12-19T19:05:03.821 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Failed to reset failed state of unit ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mgr.smithi055.dwclvp.service: Unit ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mgr.smithi055.dwclvp.service not loaded. 2023-12-19T19:05:03.984 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715.target.wants/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mgr.smithi055.dwclvp.service → /etc/systemd/system/ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@.service. 2023-12-19T19:05:04.257 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:04 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/654794312' entity='client.admin' 2023-12-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T19:05:04.469 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2023-12-19T19:05:04.477 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T19:05:04.477 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:05:04.477 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2023-12-19T19:05:04.477 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to enable service . firewalld.service is not available 2023-12-19T19:05:04.486 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T19:05:04.486 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2023-12-19T19:05:04.493 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T19:05:04.493 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:05:04.493 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2023-12-19T19:05:04.494 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-19T19:05:04.494 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr to start... 2023-12-19T19:05:04.494 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr... 2023-12-19T19:05:05.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:05 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4123863688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "63de9ba6-9ea1-11ee-95a5-87774f69a715", 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T19:05:05.683 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T19:05:05.684 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T19:05:05.685 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:05.686 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T19:04:57.837171+0000", 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.960+0000 7f6c0cf24700 1 Processor -- start 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0cf24700 1 -- start start 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0cf24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0cf24700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c08102c20 con 0x7f6c081022d0 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39004/0 (socket says 172.21.15.55:39004) 2023-12-19T19:05:05.687 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.961+0000 7f6c0659c700 1 -- 172.21.15.55:0/1149367649 learned_addr learned my addr 172.21.15.55:0/1149367649 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.962+0000 7f6c0659c700 1 -- 172.21.15.55:0/1149367649 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c08102d60 con 0x7f6c081022d0 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.962+0000 7f6c0659c700 1 --2- 172.21.15.55:0/1149367649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80098e0 tx=0x7f6bf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fa45aaf80159d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.962+0000 7f6c0559a700 1 -- 172.21.15.55:0/1149367649 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf800b860 con 0x7f6c081022d0 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.962+0000 7f6c0559a700 1 -- 172.21.15.55:0/1149367649 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6bf800be30 con 0x7f6c081022d0 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.962+0000 7f6c0559a700 1 -- 172.21.15.55:0/1149367649 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8010380 con 0x7f6c081022d0 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 -- 172.21.15.55:0/1149367649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 msgr2=0x7f6c081026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 --2- 172.21.15.55:0/1149367649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6bf80098e0 tx=0x7f6bf8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 -- 172.21.15.55:0/1149367649 shutdown_connections 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 --2- 172.21.15.55:0/1149367649 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c081026e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 -- 172.21.15.55:0/1149367649 >> 172.21.15.55:0/1149367649 conn(0x7f6c080fda50 msgr2=0x7f6c080ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 -- 172.21.15.55:0/1149367649 shutdown_connections 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 -- 172.21.15.55:0/1149367649 wait complete. 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.963+0000 7f6c0cf24700 1 Processor -- start 2023-12-19T19:05:05.688 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0cf24700 1 -- start start 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0cf24700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0cf24700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c08076080 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39014/0 (socket says 172.21.15.55:39014) 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0659c700 1 -- 172.21.15.55:0/4123863688 learned_addr learned my addr 172.21.15.55:0/4123863688 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0659c700 1 -- 172.21.15.55:0/4123863688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf80094f0 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6c0659c700 1 --2- 172.21.15.55:0/4123863688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8000c00 tx=0x7f6bf8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.964+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8010a80 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c08076280 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6bf801a430 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf80245b0 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c08076690 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6bf8024a70 con 0x7f6c081022d0 2023-12-19T19:05:05.689 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.965+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6bf802eb50 con 0x7f6c081022d0 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.966+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0804bb80 con 0x7f6c081022d0 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:04.968+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6bf801a960 con 0x7f6c081022d0 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.041+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6c081041d0 con 0x7f6c081022d0 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.042+0000 7f6bfeffd700 1 -- 172.21.15.55:0/4123863688 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f6bf802ed10 con 0x7f6c081022d0 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 msgr2=0x7f6c08077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 --2- 172.21.15.55:0/4123863688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8000c00 tx=0x7f6bf8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 shutdown_connections 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 --2- 172.21.15.55:0/4123863688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6c081022d0 0x7f6c08077c00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 >> 172.21.15.55:0/4123863688 conn(0x7f6c080fda50 msgr2=0x7f6c080ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 shutdown_connections 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:05.043+0000 7f6c0cf24700 1 -- 172.21.15.55:0/4123863688 wait complete. 2023-12-19T19:05:05.690 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (1/15)... 2023-12-19T19:05:08.750 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:08 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2297374791' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T19:05:08.767 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:08.767 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:08.767 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "63de9ba6-9ea1-11ee-95a5-87774f69a715", 2023-12-19T19:05:08.767 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:08.768 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T19:05:08.769 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T19:05:08.776 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T19:05:08.776 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T19:05:08.777 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T19:04:57.837171+0000", 2023-12-19T19:05:08.778 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ef4c6700 1 Processor -- start 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ef4c6700 1 -- start start 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ef4c6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ef4c6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8105440 con 0x7fd1e8070fa0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ed262700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ed262700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59898/0 (socket says 172.21.15.55:59898) 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.088+0000 7fd1ed262700 1 -- 172.21.15.55:0/1671455491 learned_addr learned my addr 172.21.15.55:0/1671455491 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.089+0000 7fd1ed262700 1 -- 172.21.15.55:0/1671455491 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e8105580 con 0x7fd1e8070fa0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.089+0000 7fd1ed262700 1 --2- 172.21.15.55:0/1671455491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40098e0 tx=0x7fd1e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290eedc501fd375c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.089+0000 7fd1dbfff700 1 -- 172.21.15.55:0/1671455491 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e400b860 con 0x7fd1e8070fa0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.089+0000 7fd1dbfff700 1 -- 172.21.15.55:0/1671455491 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd1e400be30 con 0x7fd1e8070fa0 2023-12-19T19:05:08.779 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.089+0000 7fd1dbfff700 1 -- 172.21.15.55:0/1671455491 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e4010380 con 0x7fd1e8070fa0 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/1671455491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 msgr2=0x7fd1e8104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 --2- 172.21.15.55:0/1671455491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1e40098e0 tx=0x7fd1e4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/1671455491 shutdown_connections 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 --2- 172.21.15.55:0/1671455491 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8104f00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/1671455491 >> 172.21.15.55:0/1671455491 conn(0x7fd1e80fc970 msgr2=0x7fd1e80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/1671455491 shutdown_connections 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/1671455491 wait complete. 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 Processor -- start 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.090+0000 7fd1ef4c6700 1 -- start start 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ef4c6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ef4c6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8100e00 con 0x7fd1e8070fa0 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ed262700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ed262700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59904/0 (socket says 172.21.15.55:59904) 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ed262700 1 -- 172.21.15.55:0/2297374791 learned_addr learned my addr 172.21.15.55:0/2297374791 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ed262700 1 -- 172.21.15.55:0/2297374791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e40094f0 con 0x7fd1e8070fa0 2023-12-19T19:05:08.780 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ed262700 1 --2- 172.21.15.55:0/2297374791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4006b00 tx=0x7fd1e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e4010a80 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd1e401a430 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e8101000 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.091+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd1e40245b0 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.092+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e8101410 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.092+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd1e4024a60 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.092+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd1e402eaa0 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.093+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e804bb80 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.095+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd1e402ed00 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.170+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd1e8106300 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.170+0000 7fd1d9ffb700 1 -- 172.21.15.55:0/2297374791 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fd1e8106300 con 0x7fd1e8070fa0 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 msgr2=0x7fd1e8102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 --2- 172.21.15.55:0/2297374791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1e4006b00 tx=0x7fd1e4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:08.781 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 shutdown_connections 2023-12-19T19:05:08.782 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 --2- 172.21.15.55:0/2297374791 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1e8070fa0 0x7fd1e8102980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:08.782 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 >> 172.21.15.55:0/2297374791 conn(0x7fd1e80fc970 msgr2=0x7fd1e80fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:08.782 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 shutdown_connections 2023-12-19T19:05:08.782 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:08.172+0000 7fd1ef4c6700 1 -- 172.21.15.55:0/2297374791 wait complete. 2023-12-19T19:05:08.782 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (2/15)... 2023-12-19T19:05:11.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: Activating manager daemon smithi055.dwclvp 2023-12-19T19:05:11.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: mgrmap e2: smithi055.dwclvp(active, starting, since 0.00285537s) 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi055.dwclvp", "id": "smithi055.dwclvp"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: Manager daemon smithi055.dwclvp is now available 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/mirror_snapshot_schedule"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/trash_purge_schedule"}]: dispatch 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:11.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:11 smithi055 ceph-mon[80978]: from='mgr.14100 172.21.15.55:0/1026368710' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:11.832 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:11.832 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "63de9ba6-9ea1-11ee-95a5-87774f69a715", 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:11.833 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T19:05:11.834 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T19:05:11.842 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T19:05:11.843 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T19:04:57.837171+0000", 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e38927700 1 Processor -- start 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e38927700 1 -- start start 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e38927700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e38927700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34105420 con 0x7f2e34076c20 2023-12-19T19:05:11.844 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59970/0 (socket says 172.21.15.55:59970) 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.164+0000 7f2e3259c700 1 -- 172.21.15.55:0/1965244529 learned_addr learned my addr 172.21.15.55:0/1965244529 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.165+0000 7f2e3259c700 1 -- 172.21.15.55:0/1965244529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e34105560 con 0x7f2e34076c20 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.165+0000 7f2e3259c700 1 --2- 172.21.15.55:0/1965244529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e240098e0 tx=0x7f2e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ab17d81ca9b872 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.165+0000 7f2e3159a700 1 -- 172.21.15.55:0/1965244529 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e2400b860 con 0x7f2e34076c20 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.165+0000 7f2e3159a700 1 -- 172.21.15.55:0/1965244529 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2e2400be30 con 0x7f2e34076c20 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.165+0000 7f2e3159a700 1 -- 172.21.15.55:0/1965244529 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e24010380 con 0x7f2e34076c20 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 -- 172.21.15.55:0/1965244529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 msgr2=0x7f2e34104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 --2- 172.21.15.55:0/1965244529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2e240098e0 tx=0x7f2e24009250 comp rx=0 tx=0).stop 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 -- 172.21.15.55:0/1965244529 shutdown_connections 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 --2- 172.21.15.55:0/1965244529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e34104ee0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 -- 172.21.15.55:0/1965244529 >> 172.21.15.55:0/1965244529 conn(0x7f2e340fc950 msgr2=0x7f2e340fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 -- 172.21.15.55:0/1965244529 shutdown_connections 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.166+0000 7f2e38927700 1 -- 172.21.15.55:0/1965244529 wait complete. 2023-12-19T19:05:11.845 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e38927700 1 Processor -- start 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e38927700 1 -- start start 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e38927700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e38927700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3410f490 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e3259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59986/0 (socket says 172.21.15.55:59986) 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e3259c700 1 -- 172.21.15.55:0/2181544200 learned_addr learned my addr 172.21.15.55:0/2181544200 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e3259c700 1 -- 172.21.15.55:0/2181544200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240094f0 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.167+0000 7f2e3259c700 1 --2- 172.21.15.55:0/2181544200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006b00 tx=0x7f2e24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e24010a80 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3410f690 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2e2401a430 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e240245b0 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3410faa0 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.168+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 2) v1 ==== 43037+0+0 (secure 0 0 0) 0x7f2e240247e0 con 0x7f2e34076c20 2023-12-19T19:05:11.846 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.169+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2e2404a680 con 0x7f2e34076c20 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.169+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3405f270 con 0x7f2e34076c20 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.172+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2e24024a90 con 0x7f2e34076c20 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.249+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2e341089b0 con 0x7f2e34076c20 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.249+0000 7f2e22ffd700 1 -- 172.21.15.55:0/2181544200 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f2e2402cb80 con 0x7f2e34076c20 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 msgr2=0x7f2e3410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 --2- 172.21.15.55:0/2181544200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2e24006b00 tx=0x7f2e24009250 comp rx=0 tx=0).stop 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 shutdown_connections 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 --2- 172.21.15.55:0/2181544200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2e34076c20 0x7f2e3410ef50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 >> 172.21.15.55:0/2181544200 conn(0x7f2e340fc950 msgr2=0x7f2e341030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 shutdown_connections 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:11.250+0000 7f2e38927700 1 -- 172.21.15.55:0/2181544200 wait complete. 2023-12-19T19:05:11.847 INFO:teuthology.orchestra.run.smithi055.stdout:mgr not available, waiting (3/15)... 2023-12-19T19:05:12.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:12 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2181544200' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T19:05:12.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:12 smithi055 ceph-mon[80978]: mgrmap e3: smithi055.dwclvp(active, since 1.00752s) 2023-12-19T19:05:14.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:14 smithi055 ceph-mon[80978]: mgrmap e4: smithi055.dwclvp(active, since 2s) 2023-12-19T19:05:17.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:16 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2583172183' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-19T19:05:23.056 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsid": "63de9ba6-9ea1-11ee-95a5-87774f69a715", 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "health": { 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 0 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "smithi055" 2023-12-19T19:05:23.057 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-19T19:05:23.058 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-19T19:05:23.059 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "restful" 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ], 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "modified": "2023-12-19T19:04:57.837171+0000", 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-19T19:05:23.060 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout }, 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.527+0000 7ff0e82fd700 1 Processor -- start 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.527+0000 7ff0e82fd700 1 -- start start 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.527+0000 7ff0e82fd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.527+0000 7ff0e82fd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e0102c00 con 0x7ff0e01022b0 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.527+0000 7ff0e6099700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e6099700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60012/0 (socket says 172.21.15.55:60012) 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e6099700 1 -- 172.21.15.55:0/1102375841 learned_addr learned my addr 172.21.15.55:0/1102375841 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e6099700 1 -- 172.21.15.55:0/1102375841 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e0102d40 con 0x7ff0e01022b0 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e6099700 1 --2- 172.21.15.55:0/1102375841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0098e0 tx=0x7ff0dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f4d717aed7db79 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:23.061 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e5097700 1 -- 172.21.15.55:0/1102375841 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc00b860 con 0x7ff0e01022b0 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e5097700 1 -- 172.21.15.55:0/1102375841 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff0dc00be30 con 0x7ff0e01022b0 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.528+0000 7ff0e5097700 1 -- 172.21.15.55:0/1102375841 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc010380 con 0x7ff0e01022b0 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 -- 172.21.15.55:0/1102375841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 msgr2=0x7ff0e01026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/1102375841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0098e0 tx=0x7ff0dc009250 comp rx=0 tx=0).stop 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 -- 172.21.15.55:0/1102375841 shutdown_connections 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/1102375841 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e01026c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 -- 172.21.15.55:0/1102375841 >> 172.21.15.55:0/1102375841 conn(0x7ff0e00fda10 msgr2=0x7ff0e00ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 -- 172.21.15.55:0/1102375841 shutdown_connections 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.529+0000 7ff0e82fd700 1 -- 172.21.15.55:0/1102375841 wait complete. 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e82fd700 1 Processor -- start 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e82fd700 1 -- start start 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e82fd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e82fd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e0111690 con 0x7ff0e01022b0 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e6099700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:23.062 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e6099700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60026/0 (socket says 172.21.15.55:60026) 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e6099700 1 -- 172.21.15.55:0/2583172183 learned_addr learned my addr 172.21.15.55:0/2583172183 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e6099700 1 -- 172.21.15.55:0/2583172183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc0094f0 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.530+0000 7ff0e6099700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006b00 tx=0x7ff0dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.531+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc0109b0 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.531+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff0dc024440 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.531+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e0111890 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.531+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0dc01a5f0 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.531+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e0111ca0 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7ff0dc010b10 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e004bb80 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0d6ffd700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 0x7ff0cc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff0dc04a540 con 0x7ff0e01022b0 2023-12-19T19:05:23.063 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0e5898700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 0x7ff0cc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.532+0000 7ff0e5898700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 0x7ff0cc038aa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009990 tx=0x7ff0d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.537+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff0dc0239f0 con 0x7ff0e01022b0 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.812+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff0e010af30 con 0x7ff0e01022b0 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.812+0000 7ff0d6ffd700 1 -- 172.21.15.55:0/2583172183 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7ff0dc0233a0 con 0x7ff0e01022b0 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 msgr2=0x7ff0cc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 0x7ff0cc038aa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0009990 tx=0x7ff0d0006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 msgr2=0x7ff0e0111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc006b00 tx=0x7ff0dc009250 comp rx=0 tx=0).stop 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 shutdown_connections 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7ff0cc0365f0 0x7ff0cc038aa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 --2- 172.21.15.55:0/2583172183 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff0e01022b0 0x7ff0e0111150 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 >> 172.21.15.55:0/2583172183 conn(0x7ff0e00fda10 msgr2=0x7ff0e00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 shutdown_connections 2023-12-19T19:05:23.064 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:16.814+0000 7ff0e82fd700 1 -- 172.21.15.55:0/2583172183 wait complete. 2023-12-19T19:05:23.065 INFO:teuthology.orchestra.run.smithi055.stdout:mgr is available 2023-12-19T19:05:24.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:24 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1515554742' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-19T19:05:24.738 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [global] 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout fsid = 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout [osd] 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-19T19:05:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02bd324700 1 Processor -- start 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02bd324700 1 -- start start 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02bd324700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02bd324700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b8104310 con 0x7f02b81039c0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59418/0 (socket says 172.21.15.55:59418) 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.014+0000 7f02b6ffd700 1 -- 172.21.15.55:0/1985483911 learned_addr learned my addr 172.21.15.55:0/1985483911 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.015+0000 7f02b6ffd700 1 -- 172.21.15.55:0/1985483911 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8101fb0 con 0x7f02b81039c0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.015+0000 7f02b6ffd700 1 --2- 172.21.15.55:0/1985483911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02a80098e0 tx=0x7f02a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37fb2753b3ba1701 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.015+0000 7f02b5ffb700 1 -- 172.21.15.55:0/1985483911 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a800b860 con 0x7f02b81039c0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.015+0000 7f02b5ffb700 1 -- 172.21.15.55:0/1985483911 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f02a800be30 con 0x7f02b81039c0 2023-12-19T19:05:24.740 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.015+0000 7f02b5ffb700 1 -- 172.21.15.55:0/1985483911 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a8010380 con 0x7f02b81039c0 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- 172.21.15.55:0/1985483911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 msgr2=0x7f02b8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 --2- 172.21.15.55:0/1985483911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f02a80098e0 tx=0x7f02a8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- 172.21.15.55:0/1985483911 shutdown_connections 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 --2- 172.21.15.55:0/1985483911 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8103dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- 172.21.15.55:0/1985483911 >> 172.21.15.55:0/1985483911 conn(0x7f02b80fda40 msgr2=0x7f02b80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- 172.21.15.55:0/1985483911 shutdown_connections 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- 172.21.15.55:0/1985483911 wait complete. 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 Processor -- start 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.016+0000 7f02bd324700 1 -- start start 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02bd324700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02bd324700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b8077190 con 0x7f02b81039c0 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:24.741 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02b6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59424/0 (socket says 172.21.15.55:59424) 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02b6ffd700 1 -- 172.21.15.55:0/1515554742 learned_addr learned my addr 172.21.15.55:0/1515554742 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02b6ffd700 1 -- 172.21.15.55:0/1515554742 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80094f0 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f02b6ffd700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c00 tx=0x7f02a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.017+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a8010a80 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.018+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f02a801a430 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.018+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b8077390 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.018+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02a80245b0 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.018+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b80777a0 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.018+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f02a8024820 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.019+0000 7f029f7fe700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 0x7f02a0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.019+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f02a804a6d0 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.019+0000 7f02b67fc700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 0x7f02a0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.019+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b805f270 con 0x7f02b81039c0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.019+0000 7f02b67fc700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 0x7f02a0038aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009990 tx=0x7f02ac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:24.742 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.024+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f02a8018530 con 0x7f02b81039c0 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.219+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f02b804bbb0 con 0x7f02b81039c0 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.220+0000 7f029f7fe700 1 -- 172.21.15.55:0/1515554742 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f02a8018350 con 0x7f02b81039c0 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 msgr2=0x7f02a0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 0x7f02a0038aa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009990 tx=0x7f02ac006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 msgr2=0x7f02b8078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c00 tx=0x7f02a8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 shutdown_connections 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f02a00365f0 0x7f02a0038aa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 --2- 172.21.15.55:0/1515554742 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f02b81039c0 0x7f02b8078d10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 >> 172.21.15.55:0/1515554742 conn(0x7f02b80fda40 msgr2=0x7f02b80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 shutdown_connections 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:24.222+0000 7f02bd324700 1 -- 172.21.15.55:0/1515554742 wait complete. 2023-12-19T19:05:24.743 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling cephadm module... 2023-12-19T19:05:25.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:25 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2938168509' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-19T19:05:26.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.224+0000 7f17f80f2700 1 Processor -- start 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.224+0000 7f17f80f2700 1 -- start start 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f80f2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f80f2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0102c20 con 0x7f17f01022d0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f5e8e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f5e8e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59438/0 (socket says 172.21.15.55:59438) 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f5e8e700 1 -- 172.21.15.55:0/2764734819 learned_addr learned my addr 172.21.15.55:0/2764734819 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f5e8e700 1 -- 172.21.15.55:0/2764734819 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0102d60 con 0x7f17f01022d0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f5e8e700 1 --2- 172.21.15.55:0/2764734819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cecb412bc938ee1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.225+0000 7f17f4e8c700 1 -- 172.21.15.55:0/2764734819 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e400b860 con 0x7f17f01022d0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f4e8c700 1 -- 172.21.15.55:0/2764734819 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f17e400be30 con 0x7f17f01022d0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f4e8c700 1 -- 172.21.15.55:0/2764734819 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4010380 con 0x7f17f01022d0 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 -- 172.21.15.55:0/2764734819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 msgr2=0x7f17f01026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2764734819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:26.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 -- 172.21.15.55:0/2764734819 shutdown_connections 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2764734819 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f01026e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 -- 172.21.15.55:0/2764734819 >> 172.21.15.55:0/2764734819 conn(0x7f17f00fda50 msgr2=0x7f17f00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 -- 172.21.15.55:0/2764734819 shutdown_connections 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.226+0000 7f17f80f2700 1 -- 172.21.15.55:0/2764734819 wait complete. 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f80f2700 1 Processor -- start 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f80f2700 1 -- start start 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f80f2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f80f2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f0076080 con 0x7f17f01022d0 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f5e8e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f5e8e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59444/0 (socket says 172.21.15.55:59444) 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f5e8e700 1 -- 172.21.15.55:0/2938168509 learned_addr learned my addr 172.21.15.55:0/2938168509 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.227+0000 7f17f5e8e700 1 -- 172.21.15.55:0/2938168509 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40094f0 con 0x7f17f01022d0 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17f5e8e700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006b00 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e40109b0 con 0x7f17f01022d0 2023-12-19T19:05:26.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f17e4024440 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f0076280 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e401a5f0 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.228+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f0076690 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f17e4024990 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f004bb80 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17e27fc700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 0x7f17dc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f17e404a520 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17f568d700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 0x7f17dc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.229+0000 7f17f568d700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 0x7f17dc038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009990 tx=0x7f17ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.234+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f17e402d3f0 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.484+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f17f01040e0 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.537+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f17e400fda0 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.538+0000 7f17e27fc700 1 -- 172.21.15.55:0/2938168509 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f17e4018350 con 0x7f17f01022d0 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 msgr2=0x7f17dc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:26.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 0x7f17dc038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009990 tx=0x7f17ec006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 msgr2=0x7f17f0077c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17e4006b00 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 shutdown_connections 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f17dc0365f0 0x7f17dc038aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 --2- 172.21.15.55:0/2938168509 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f17f01022d0 0x7f17f0077c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.540+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 >> 172.21.15.55:0/2938168509 conn(0x7f17f00fda50 msgr2=0x7f17f00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.541+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 shutdown_connections 2023-12-19T19:05:26.106 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:25.541+0000 7f17f80f2700 1 -- 172.21.15.55:0/2938168509 wait complete. 2023-12-19T19:05:26.711 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:26 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2938168509' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-19T19:05:26.711 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:26 smithi055 ceph-mon[80978]: mgrmap e5: smithi055.dwclvp(active, since 14s) 2023-12-19T19:05:27.336 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "smithi055.dwclvp", 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f11e1c700 1 Processor -- start 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f11e1c700 1 -- start start 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f11e1c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f11e1c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c102c00 con 0x7f9f0c1022b0 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59462/0 (socket says 172.21.15.55:59462) 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f0b7fe700 1 -- 172.21.15.55:0/182808544 learned_addr learned my addr 172.21.15.55:0/182808544 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:27.337 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.503+0000 7f9f0b7fe700 1 -- 172.21.15.55:0/182808544 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c102d40 con 0x7f9f0c1022b0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f0b7fe700 1 --2- 172.21.15.55:0/182808544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0098e0 tx=0x7f9efc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b13b7ab6abdcee server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f0a7fc700 1 -- 172.21.15.55:0/182808544 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9efc00b860 con 0x7f9f0c1022b0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f0a7fc700 1 -- 172.21.15.55:0/182808544 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9efc00be30 con 0x7f9f0c1022b0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f0a7fc700 1 -- 172.21.15.55:0/182808544 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9efc010380 con 0x7f9f0c1022b0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f11e1c700 1 -- 172.21.15.55:0/182808544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 msgr2=0x7f9f0c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/182808544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0098e0 tx=0x7f9efc009250 comp rx=0 tx=0).stop 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f11e1c700 1 -- 172.21.15.55:0/182808544 shutdown_connections 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/182808544 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c1026c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.504+0000 7f9f11e1c700 1 -- 172.21.15.55:0/182808544 >> 172.21.15.55:0/182808544 conn(0x7f9f0c0fda10 msgr2=0x7f9f0c0ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 -- 172.21.15.55:0/182808544 shutdown_connections 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 -- 172.21.15.55:0/182808544 wait complete. 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 Processor -- start 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 -- start start 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.505+0000 7f9f11e1c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0c077190 con 0x7f9f0c1022b0 2023-12-19T19:05:27.338 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f0b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:59478/0 (socket says 172.21.15.55:59478) 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f0b7fe700 1 -- 172.21.15.55:0/1644151193 learned_addr learned my addr 172.21.15.55:0/1644151193 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f0b7fe700 1 -- 172.21.15.55:0/1644151193 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0094f0 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f0b7fe700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9efc000c00 tx=0x7f9efc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9efc010a80 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9efc01a430 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c077390 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9efc0245b0 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.506+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c0777a0 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.507+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f9efc0247a0 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.507+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c04bb80 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.507+0000 7f9ef3fff700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 0x7f9ef4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.507+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9efc04ba90 con 0x7f9f0c1022b0 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.508+0000 7f9f0affd700 1 -- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 msgr2=0x7f9ef4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:27.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.508+0000 7f9f0affd700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 0x7f9ef4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.512+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9efc018350 con 0x7f9f0c1022b0 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.709+0000 7f9f0affd700 1 -- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 msgr2=0x7f9ef4038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.709+0000 7f9f0affd700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 0x7f9ef4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.762+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9f0c10af20 con 0x7f9f0c1022b0 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.763+0000 7f9ef3fff700 1 -- 172.21.15.55:0/1644151193 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9efc023b80 con 0x7f9f0c1022b0 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 msgr2=0x7f9ef4038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 0x7f9ef4038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 msgr2=0x7f9f0c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9efc000c00 tx=0x7f9efc009250 comp rx=0 tx=0).stop 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 shutdown_connections 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f9ef4036640 0x7f9ef4038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 --2- 172.21.15.55:0/1644151193 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f9f0c1022b0 0x7f9f0c078d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 >> 172.21.15.55:0/1644151193 conn(0x7f9f0c0fda10 msgr2=0x7f9f0c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 shutdown_connections 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:26.765+0000 7f9f11e1c700 1 -- 172.21.15.55:0/1644151193 wait complete. 2023-12-19T19:05:27.340 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2023-12-19T19:05:27.341 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 5... 2023-12-19T19:05:27.639 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:27 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1644151193' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-19T19:05:32.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: Active manager daemon smithi055.dwclvp restarted 2023-12-19T19:05:32.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: Activating manager daemon smithi055.dwclvp 2023-12-19T19:05:32.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: osdmap e2: 0 total, 0 up, 0 in 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: mgrmap e6: smithi055.dwclvp(active, starting, since 0.00571825s) 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi055.dwclvp", "id": "smithi055.dwclvp"}]: dispatch 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: Manager daemon smithi055.dwclvp is now available 2023-12-19T19:05:32.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:32.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:32.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:32.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:32 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:33.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:33 smithi055 ceph-mon[80978]: Found migration_current of "None". Setting to last migration. 2023-12-19T19:05:33.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:33 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:33.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:33 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/mirror_snapshot_schedule"}]: dispatch 2023-12-19T19:05:33.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:33 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/trash_purge_schedule"}]: dispatch 2023-12-19T19:05:33.660 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:33 smithi055 ceph-mon[80978]: mgrmap e7: smithi055.dwclvp(active, since 1.00911s) 2023-12-19T19:05:33.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:33.954 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-12-19T19:05:33.954 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-19T19:05:33.954 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.736+0000 7f82fe6ec700 1 Processor -- start 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82fe6ec700 1 -- start start 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82fe6ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82fe6ec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8101b10 con 0x7f82f81011c0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39050/0 (socket says 172.21.15.55:39050) 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82f7fff700 1 -- 172.21.15.55:0/4249076580 learned_addr learned my addr 172.21.15.55:0/4249076580 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82f7fff700 1 -- 172.21.15.55:0/4249076580 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f8101c50 con 0x7f82f81011c0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.737+0000 7f82f7fff700 1 --2- 172.21.15.55:0/4249076580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82e80098e0 tx=0x7f82e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251fab856fed8e78 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82f6ffd700 1 -- 172.21.15.55:0/4249076580 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e800b860 con 0x7f82f81011c0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82f6ffd700 1 -- 172.21.15.55:0/4249076580 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f82e800be30 con 0x7f82f81011c0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82f6ffd700 1 -- 172.21.15.55:0/4249076580 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e8010380 con 0x7f82f81011c0 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4249076580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 msgr2=0x7f82f81015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4249076580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82e80098e0 tx=0x7f82e8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:33.955 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4249076580 shutdown_connections 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4249076580 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f81015d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4249076580 >> 172.21.15.55:0/4249076580 conn(0x7f82f80fc980 msgr2=0x7f82f80fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4249076580 shutdown_connections 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.738+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4249076580 wait complete. 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82fe6ec700 1 Processor -- start 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82fe6ec700 1 -- start start 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82fe6ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82fe6ec700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81059e0 con 0x7f82f81011c0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82f7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39064/0 (socket says 172.21.15.55:39064) 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.739+0000 7f82f7fff700 1 -- 172.21.15.55:0/4140756602 learned_addr learned my addr 172.21.15.55:0/4140756602 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82f7fff700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80094f0 con 0x7f82f81011c0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82f7fff700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82e8006b00 tx=0x7f82e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e80109b0 con 0x7f82f81011c0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f8105be0 con 0x7f82f81011c0 2023-12-19T19:05:33.956 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f82e8024440 con 0x7f82f81011c0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82e801a5f0 con 0x7f82f81011c0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.740+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f8105ff0 con 0x7f82f81011c0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f82e801a8b0 con 0x7f82f81011c0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f4ff9700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e00391b0 con 0x7f82e00365f0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f82e804a610 con 0x7f82f81011c0 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f77fe700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.741+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.942+0000 7f82f77fe700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:27.942+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:28.343+0000 7f82f77fe700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:28.343+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:29.144+0000 7f82f77fe700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:33.957 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:29.144+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:30.745+0000 7f82f77fe700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/917343204 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:30.745+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:32.361+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7f82e800fdc0 con 0x7f82f81011c0 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:32.361+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 msgr2=0x7f82e0038aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:32.361+0000 7f82f4ff9700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.363+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f82e803aa40 con 0x7f82f81011c0 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.363+0000 7f82f4ff9700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 0x7f82e003c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.363+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e00391b0 con 0x7f82e0039f60 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.364+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 0x7f82e003c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.365+0000 7f82f77fe700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 0x7f82e003c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0039d0 tx=0x7f82ec009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.365+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f82e00391b0 con 0x7f82e0039f60 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f82f805f270 con 0x7f82e0039f60 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82f4ff9700 1 -- 172.21.15.55:0/4140756602 <== mgr.14120 v2:172.21.15.55:6800/2922737294 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f82f805f270 con 0x7f82e0039f60 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 msgr2=0x7f82e003c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:33.958 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 0x7f82e003c340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0039d0 tx=0x7f82ec009350 comp rx=0 tx=0).stop 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 msgr2=0x7f82f8107560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.372+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f82e8006b00 tx=0x7f82e8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 shutdown_connections 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f82e0039f60 0x7f82e003c340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f82f81011c0 0x7f82f8107560 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 --2- 172.21.15.55:0/4140756602 >> [v2:172.21.15.55:6800/917343204,v1:172.21.15.55:6801/917343204] conn(0x7f82e00365f0 0x7f82e0038aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 >> 172.21.15.55:0/4140756602 conn(0x7f82f80fc980 msgr2=0x7f82f80fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 shutdown_connections 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:33.373+0000 7f82fe6ec700 1 -- 172.21.15.55:0/4140756602 wait complete. 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 5 is available 2023-12-19T19:05:33.959 INFO:teuthology.orchestra.run.smithi055.stdout:Setting orchestrator backend to cephadm... 2023-12-19T19:05:34.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:34 smithi055 ceph-mon[80978]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-19T19:05:34.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:34 smithi055 ceph-mon[80978]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-19T19:05:34.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:34 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:34.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:34 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.364+0000 7fa4c4d27700 1 Processor -- start 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.364+0000 7fa4c4d27700 1 -- start start 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.364+0000 7fa4c4d27700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4c4d27700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01054c0 con 0x7fa4c0070f20 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4be59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4be59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39126/0 (socket says 172.21.15.55:39126) 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4be59c700 1 -- 172.21.15.55:0/3633698690 learned_addr learned my addr 172.21.15.55:0/3633698690 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:35.101 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4be59c700 1 -- 172.21.15.55:0/3633698690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0105600 con 0x7fa4c0070f20 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4be59c700 1 --2- 172.21.15.55:0/3633698690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009e50 tx=0x7fa4b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958149c8e0239400 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4bd59a700 1 -- 172.21.15.55:0/3633698690 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b00092e0 con 0x7fa4c0070f20 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4bd59a700 1 -- 172.21.15.55:0/3633698690 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa4b000be90 con 0x7fa4c0070f20 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.365+0000 7fa4bd59a700 1 -- 172.21.15.55:0/3633698690 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b0007400 con 0x7fa4c0070f20 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3633698690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 msgr2=0x7fa4c0104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3633698690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0009e50 tx=0x7fa4b0009840 comp rx=0 tx=0).stop 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3633698690 shutdown_connections 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3633698690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0104f80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3633698690 >> 172.21.15.55:0/3633698690 conn(0x7fa4c00fc970 msgr2=0x7fa4c00fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3633698690 shutdown_connections 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.366+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3633698690 wait complete. 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4c4d27700 1 Processor -- start 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4c4d27700 1 -- start start 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4c4d27700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:35.102 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4c4d27700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c0100e80 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4be59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4be59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39136/0 (socket says 172.21.15.55:39136) 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4be59c700 1 -- 172.21.15.55:0/3582023620 learned_addr learned my addr 172.21.15.55:0/3582023620 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4be59c700 1 -- 172.21.15.55:0/3582023620 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0009ae0 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.367+0000 7fa4be59c700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0000f80 tx=0x7fa4b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b001c620 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa4b001cbf0 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4b0007b40 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c0101080 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c0101430 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.368+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fa4b0019070 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.369+0000 7fa4b6ffd700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 0x7fa4ac0389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.369+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa4b0049d30 con 0x7fa4c0070f20 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.369+0000 7fa4bdd9b700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 0x7fa4ac0389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:35.103 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.369+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c010b140 con 0x7fa4c0070f20 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.369+0000 7fa4bdd9b700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 0x7fa4ac0389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009990 tx=0x7fa4a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.374+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa4b0017080 con 0x7fa4c0070f20 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.583+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4c01016d0 con 0x7fa4ac036510 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.590+0000 7fa4b6ffd700 1 -- 172.21.15.55:0/3582023620 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa4c01016d0 con 0x7fa4ac036510 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 msgr2=0x7fa4ac0389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 0x7fa4ac0389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8009990 tx=0x7fa4a8006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 msgr2=0x7fa4c0102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0000f80 tx=0x7fa4b0009840 comp rx=0 tx=0).stop 2023-12-19T19:05:35.104 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 shutdown_connections 2023-12-19T19:05:35.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa4ac036510 0x7fa4ac0389c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:35.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 --2- 172.21.15.55:0/3582023620 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa4c0070f20 0x7fa4c0102a00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:35.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.592+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 >> 172.21.15.55:0/3582023620 conn(0x7fa4c00fc970 msgr2=0x7fa4c00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:35.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.593+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 shutdown_connections 2023-12-19T19:05:35.105 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:34.593+0000 7fa4c4d27700 1 -- 172.21.15.55:0/3582023620 wait complete. 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: [19/Dec/2023:19:05:33] ENGINE Bus STARTING 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: [19/Dec/2023:19:05:33] ENGINE Serving on https://172.21.15.55:7150 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: [19/Dec/2023:19:05:33] ENGINE Bus STARTED 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:35.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:35 smithi055 ceph-mon[80978]: mgrmap e8: smithi055.dwclvp(active, since 2s) 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout value unchanged 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.485+0000 7f71ce25f700 1 Processor -- start 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.485+0000 7f71ce25f700 1 -- start start 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.485+0000 7f71ce25f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.485+0000 7f71ce25f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8104310 con 0x7f71c81039c0 2023-12-19T19:05:36.225 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39140/0 (socket says 172.21.15.55:39140) 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c77fe700 1 -- 172.21.15.55:0/3756167507 learned_addr learned my addr 172.21.15.55:0/3756167507 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c77fe700 1 -- 172.21.15.55:0/3756167507 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8101fb0 con 0x7f71c81039c0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c77fe700 1 --2- 172.21.15.55:0/3756167507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b80098e0 tx=0x7f71b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e85f005dcd7c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c67fc700 1 -- 172.21.15.55:0/3756167507 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b800b860 con 0x7f71c81039c0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c67fc700 1 -- 172.21.15.55:0/3756167507 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f71b800be30 con 0x7f71c81039c0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.486+0000 7f71c67fc700 1 -- 172.21.15.55:0/3756167507 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b8010380 con 0x7f71c81039c0 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 -- 172.21.15.55:0/3756167507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 msgr2=0x7f71c8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 --2- 172.21.15.55:0/3756167507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b80098e0 tx=0x7f71b8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 -- 172.21.15.55:0/3756167507 shutdown_connections 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 --2- 172.21.15.55:0/3756167507 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c8103dd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 -- 172.21.15.55:0/3756167507 >> 172.21.15.55:0/3756167507 conn(0x7f71c80fda40 msgr2=0x7f71c80ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 -- 172.21.15.55:0/3756167507 shutdown_connections 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.487+0000 7f71ce25f700 1 -- 172.21.15.55:0/3756167507 wait complete. 2023-12-19T19:05:36.226 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71ce25f700 1 Processor -- start 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71ce25f700 1 -- start start 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71ce25f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71ce25f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8111630 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39152/0 (socket says 172.21.15.55:39152) 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71c77fe700 1 -- 172.21.15.55:0/693268792 learned_addr learned my addr 172.21.15.55:0/693268792 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.488+0000 7f71c77fe700 1 -- 172.21.15.55:0/693268792 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b80094f0 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71c77fe700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006b00 tx=0x7f71b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b80109b0 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f71b801a430 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8111830 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b80245b0 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.489+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8111c40 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f71b8024830 con 0x7f71c81039c0 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71affff700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 0x7f71b0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:36.227 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f71b804abd0 con 0x7f71c81039c0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c8102d60 con 0x7f71c81039c0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71c6ffd700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 0x7f71b0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.490+0000 7f71c6ffd700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 0x7f71b0038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009990 tx=0x7f71bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.495+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f71b8024b80 con 0x7f71c81039c0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.691+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f71c810afe0 con 0x7f71b00365f0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.691+0000 7f71affff700 1 -- 172.21.15.55:0/693268792 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f71c810afe0 con 0x7f71b00365f0 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.693+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 msgr2=0x7f71b0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.693+0000 7f71ce25f700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 0x7f71b0038aa0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009990 tx=0x7f71bc006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.693+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 msgr2=0x7f71c81110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.693+0000 7f71ce25f700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71b8006b00 tx=0x7f71b8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 shutdown_connections 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f71b00365f0 0x7f71b0038aa0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 --2- 172.21.15.55:0/693268792 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f71c81039c0 0x7f71c81110f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 >> 172.21.15.55:0/693268792 conn(0x7f71c80fda40 msgr2=0x7f71c80ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:36.228 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 shutdown_connections 2023-12-19T19:05:36.229 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:35.694+0000 7f71ce25f700 1 -- 172.21.15.55:0/693268792 wait complete. 2023-12-19T19:05:36.229 INFO:teuthology.orchestra.run.smithi055.stdout:Generating ssh key... 2023-12-19T19:05:36.788 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:36 smithi055 ceph-mon[80978]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:37.458 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:37 smithi055 ceph-mon[80978]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:37.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:37 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:37.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:37 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:37.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.579+0000 7f22b8bcb700 1 Processor -- start 2023-12-19T19:05:37.467 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.579+0000 7f22b8bcb700 1 -- start start 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b8bcb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b8bcb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b41053e0 con 0x7f22b4076c60 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39154/0 (socket says 172.21.15.55:39154) 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b259c700 1 -- 172.21.15.55:0/3540471157 learned_addr learned my addr 172.21.15.55:0/3540471157 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b259c700 1 -- 172.21.15.55:0/3540471157 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b4105520 con 0x7f22b4076c60 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.580+0000 7f22b259c700 1 --2- 172.21.15.55:0/3540471157 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f22a40098e0 tx=0x7f22a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffb2968a4ccd0f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b159a700 1 -- 172.21.15.55:0/3540471157 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22a400b860 con 0x7f22b4076c60 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b159a700 1 -- 172.21.15.55:0/3540471157 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f22a400be30 con 0x7f22b4076c60 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b159a700 1 -- 172.21.15.55:0/3540471157 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22a4010380 con 0x7f22b4076c60 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 -- 172.21.15.55:0/3540471157 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 msgr2=0x7f22b4104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/3540471157 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f22a40098e0 tx=0x7f22a4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 -- 172.21.15.55:0/3540471157 shutdown_connections 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/3540471157 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4104ea0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:37.468 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 -- 172.21.15.55:0/3540471157 >> 172.21.15.55:0/3540471157 conn(0x7f22b40fc950 msgr2=0x7f22b40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 -- 172.21.15.55:0/3540471157 shutdown_connections 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.581+0000 7f22b8bcb700 1 -- 172.21.15.55:0/3540471157 wait complete. 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b8bcb700 1 Processor -- start 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b8bcb700 1 -- start start 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b8bcb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b8bcb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b4111690 con 0x7f22b4076c60 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39166/0 (socket says 172.21.15.55:39166) 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b259c700 1 -- 172.21.15.55:0/2089774834 learned_addr learned my addr 172.21.15.55:0/2089774834 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.582+0000 7f22b259c700 1 -- 172.21.15.55:0/2089774834 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40094f0 con 0x7f22b4076c60 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22b259c700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22a4006b00 tx=0x7f22a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22a4010a80 con 0x7f22b4076c60 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f22a401a430 con 0x7f22b4076c60 2023-12-19T19:05:37.469 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b4111890 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22a40245b0 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.583+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b4111ca0 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.584+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f22a4010be0 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.584+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b404bb80 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.584+0000 7f22aaffd700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 0x7f22a0038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.584+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f22a404aaf0 con 0x7f22b4076c60 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.584+0000 7f22b1d9b700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 0x7f22a0038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:37.470 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.585+0000 7f22b1d9b700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 0x7f22a0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f229c009990 tx=0x7f229c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:37.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.589+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f22a40239e0 con 0x7f22b4076c60 2023-12-19T19:05:37.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.786+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f22b410afb0 con 0x7f22a00365f0 2023-12-19T19:05:37.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.898+0000 7f22aaffd700 1 -- 172.21.15.55:0/2089774834 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f22b410afb0 con 0x7f22a00365f0 2023-12-19T19:05:37.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 msgr2=0x7f22a0038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:37.471 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 0x7f22a0038aa0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f229c009990 tx=0x7f229c006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 msgr2=0x7f22b4111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f22a4006b00 tx=0x7f22a4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 shutdown_connections 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f22a00365f0 0x7f22a0038aa0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 --2- 172.21.15.55:0/2089774834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22b4076c60 0x7f22b4111150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.900+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 >> 172.21.15.55:0/2089774834 conn(0x7f22b40fc950 msgr2=0x7f22b40fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.901+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 shutdown_connections 2023-12-19T19:05:37.472 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:36.901+0000 7f22b8bcb700 1 -- 172.21.15.55:0/2089774834 wait complete. 2023-12-19T19:05:38.641 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:38 smithi055 ceph-mon[80978]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:38.641 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:38 smithi055 ceph-mon[80978]: Generating ssh key... 2023-12-19T19:05:38.648 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZuGUl201Ipp4fIlxXhtPA4TLZy2TdN6TqKMc9rmWwmshCnBqdPRdT1FTJp3DpnwxhIVUpYHF946n30FZtd3T0cTtxfpZzUVoMQGjd6i3TTyz4xwdqncwuGsw5q+SSsf6bq3OmLGLkzljbyCT8YcPv6rfeO63W7FyA186KCz62w2dQTJVb+zhYnEFjkUA0iJY7mtN8lPsJjjDnm0FtTUcCkVu/PIKM4Sgh+qHyJHAPwh+hDW40HoPSDpdOIkw3Ec++d78m7jwo1QTTX3CaCoSK/+0AS2W6InNoTQej4HiXk5/ktgZULauH6Vsn9wVxwF4uOl3FY5t+bVHLSBRn0MWEjfzu2sWaO2vQkEkQhb7xKg9i3zUG9KBMg7XlD19payeUpoMZpWoFzNeDQ+ir1bVDFV1gKcOlNWlKbYInDTEjEmjfxmSGsf2fZoKl33dUmSTOslQQKfjC/EbmWa4O0OX/90Xb66+mfWlUTmp22+5+9lGNbOA3gfFsLmCwkvNteM= ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:05:38.648 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef930b700 1 Processor -- start 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef930b700 1 -- start start 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef930b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef930b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef41053e0 con 0x7fdef4076c60 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58880/0 (socket says 172.21.15.55:58880) 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.859+0000 7fdef2ffd700 1 -- 172.21.15.55:0/3879761200 learned_addr learned my addr 172.21.15.55:0/3879761200 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.860+0000 7fdef2ffd700 1 -- 172.21.15.55:0/3879761200 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4105520 con 0x7fdef4076c60 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.860+0000 7fdef2ffd700 1 --2- 172.21.15.55:0/3879761200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77817495ca4d5aa server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.860+0000 7fdef1ffb700 1 -- 172.21.15.55:0/3879761200 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee400b860 con 0x7fdef4076c60 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.860+0000 7fdef1ffb700 1 -- 172.21.15.55:0/3879761200 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdee400be30 con 0x7fdef4076c60 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.860+0000 7fdef1ffb700 1 -- 172.21.15.55:0/3879761200 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee4010380 con 0x7fdef4076c60 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 -- 172.21.15.55:0/3879761200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 msgr2=0x7fdef4104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 --2- 172.21.15.55:0/3879761200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdee40098e0 tx=0x7fdee4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 -- 172.21.15.55:0/3879761200 shutdown_connections 2023-12-19T19:05:38.649 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 --2- 172.21.15.55:0/3879761200 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4104ea0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 -- 172.21.15.55:0/3879761200 >> 172.21.15.55:0/3879761200 conn(0x7fdef40fc950 msgr2=0x7fdef40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 -- 172.21.15.55:0/3879761200 shutdown_connections 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 -- 172.21.15.55:0/3879761200 wait complete. 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.861+0000 7fdef930b700 1 Processor -- start 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef930b700 1 -- start start 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef930b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef930b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4111690 con 0x7fdef4076c60 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef2ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58894/0 (socket says 172.21.15.55:58894) 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef2ffd700 1 -- 172.21.15.55:0/173483721 learned_addr learned my addr 172.21.15.55:0/173483721 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef2ffd700 1 -- 172.21.15.55:0/173483721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40094f0 con 0x7fdef4076c60 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdef2ffd700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006b00 tx=0x7fdee4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.862+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee4010a80 con 0x7fdef4076c60 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.863+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdee401a430 con 0x7fdef4076c60 2023-12-19T19:05:38.650 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.863+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef4111890 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.863+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef4111ca0 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.863+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdee40245b0 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.863+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fdee4024830 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.864+0000 7fdedb7fe700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 0x7fdedc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.864+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef404bb80 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.864+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdee404acb0 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.864+0000 7fdef27fc700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 0x7fdedc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.864+0000 7fdef27fc700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 0x7fdedc038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009990 tx=0x7fdee8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:37.869+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdee4018350 con 0x7fdef4076c60 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.069+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef40008d0 con 0x7fdedc0365f0 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.070+0000 7fdedb7fe700 1 -- 172.21.15.55:0/173483721 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fdef40008d0 con 0x7fdedc0365f0 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 msgr2=0x7fdedc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 0x7fdedc038aa0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009990 tx=0x7fdee8006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 msgr2=0x7fdef4111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:38.651 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006b00 tx=0x7fdee4009250 comp rx=0 tx=0).stop 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 shutdown_connections 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdedc0365f0 0x7fdedc038aa0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 --2- 172.21.15.55:0/173483721 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdef4076c60 0x7fdef4111150 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 >> 172.21.15.55:0/173483721 conn(0x7fdef40fc950 msgr2=0x7fdef40fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 shutdown_connections 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:38.072+0000 7fdef930b700 1 -- 172.21.15.55:0/173483721 wait complete. 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:Adding key to root@localhost authorized_keys... 2023-12-19T19:05:38.652 INFO:teuthology.orchestra.run.smithi055.stdout:Adding host smithi055... 2023-12-19T19:05:39.797 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:39 smithi055 ceph-mon[80978]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:40 smithi055 ceph-mon[80978]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:41.683 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:41 smithi055 ceph-mon[80978]: Deploying cephadm binary to smithi055 2023-12-19T19:05:41.683 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:41 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:41.683 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:41 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Added host 'smithi055' with addr '172.21.15.55' 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.058+0000 7fa2d56ed700 1 Processor -- start 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.058+0000 7fa2d56ed700 1 -- start start 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.058+0000 7fa2d56ed700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.058+0000 7fa2d56ed700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d01042c0 con 0x7fa2d0103970 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.058+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58898/0 (socket says 172.21.15.55:58898) 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2ceffd700 1 -- 172.21.15.55:0/1722084897 learned_addr learned my addr 172.21.15.55:0/1722084897 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:41.934 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2ceffd700 1 -- 172.21.15.55:0/1722084897 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0104400 con 0x7fa2d0103970 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2ceffd700 1 --2- 172.21.15.55:0/1722084897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1edf9a835a05d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2cdffb700 1 -- 172.21.15.55:0/1722084897 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c80092e0 con 0x7fa2d0103970 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2cdffb700 1 -- 172.21.15.55:0/1722084897 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa2c800be90 con 0x7fa2d0103970 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.059+0000 7fa2cdffb700 1 -- 172.21.15.55:0/1722084897 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c8007400 con 0x7fa2d0103970 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 -- 172.21.15.55:0/1722084897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 msgr2=0x7fa2d0103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/1722084897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c8009840 comp rx=0 tx=0).stop 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 -- 172.21.15.55:0/1722084897 shutdown_connections 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/1722084897 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d0103d80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 -- 172.21.15.55:0/1722084897 >> 172.21.15.55:0/1722084897 conn(0x7fa2d00fc0b0 msgr2=0x7fa2d00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 -- 172.21.15.55:0/1722084897 shutdown_connections 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 -- 172.21.15.55:0/1722084897 wait complete. 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.060+0000 7fa2d56ed700 1 Processor -- start 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2d56ed700 1 -- start start 2023-12-19T19:05:41.935 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2d56ed700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2d56ed700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0101630 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2ceffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58906/0 (socket says 172.21.15.55:58906) 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2ceffd700 1 -- 172.21.15.55:0/2433586594 learned_addr learned my addr 172.21.15.55:0/2433586594 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2ceffd700 1 -- 172.21.15.55:0/2433586594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8009ae0 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2ceffd700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009fd0 tx=0x7fa2c8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c801c620 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa2c801cbf0 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.061+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d0101830 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.062+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2c8007bd0 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.062+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0101ca0 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.062+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa2c8019070 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.062+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d010b150 con 0x7fa2d0103970 2023-12-19T19:05:41.936 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.063+0000 7fa2c77fe700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 0x7fa2bc038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.063+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa2c8049e60 con 0x7fa2d0103970 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.063+0000 7fa2ce7fc700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 0x7fa2bc038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.063+0000 7fa2ce7fc700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 0x7fa2bc038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009990 tx=0x7fa2b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.067+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa2c8019ac0 con 0x7fa2d0103970 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:39.266+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi055", "addr": "172.21.15.55", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2d0101fa0 con 0x7fa2bc0365f0 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.349+0000 7fa2c77fe700 1 -- 172.21.15.55:0/2433586594 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa2d0101fa0 con 0x7fa2bc0365f0 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 msgr2=0x7fa2bc038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 0x7fa2bc038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa2b8009990 tx=0x7fa2b8006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 msgr2=0x7fa2d01031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009fd0 tx=0x7fa2c8009840 comp rx=0 tx=0).stop 2023-12-19T19:05:41.937 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 shutdown_connections 2023-12-19T19:05:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa2bc0365f0 0x7fa2bc038aa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 --2- 172.21.15.55:0/2433586594 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa2d0103970 0x7fa2d01031b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.351+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 >> 172.21.15.55:0/2433586594 conn(0x7fa2d00fc0b0 msgr2=0x7fa2d00fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.352+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 shutdown_connections 2023-12-19T19:05:41.938 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:41.352+0000 7fa2d56ed700 1 -- 172.21.15.55:0/2433586594 wait complete. 2023-12-19T19:05:41.940 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying mon service with default placement... 2023-12-19T19:05:42.589 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:42 smithi055 ceph-mon[80978]: Added host smithi055 2023-12-19T19:05:43.758 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:43 smithi055 ceph-mon[80978]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:43.758 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:43 smithi055 ceph-mon[80978]: Saving service mon spec with placement count:5 2023-12-19T19:05:43.758 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:43 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.351+0000 7fb241be2700 1 Processor -- start 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.351+0000 7fb241be2700 1 -- start start 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.351+0000 7fb241be2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.351+0000 7fb241be2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c102d10 con 0x7fb23c070f80 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:44.126 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58908/0 (socket says 172.21.15.55:58908) 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23b7fe700 1 -- 172.21.15.55:0/2036533428 learned_addr learned my addr 172.21.15.55:0/2036533428 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23b7fe700 1 -- 172.21.15.55:0/2036533428 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c107860 con 0x7fb23c070f80 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23b7fe700 1 --2- 172.21.15.55:0/2036533428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d06c09a58831b96 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23a7fc700 1 -- 172.21.15.55:0/2036533428 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb22c0092e0 con 0x7fb23c070f80 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23a7fc700 1 -- 172.21.15.55:0/2036533428 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb22c00be90 con 0x7fb23c070f80 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.352+0000 7fb23a7fc700 1 -- 172.21.15.55:0/2036533428 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb22c007400 con 0x7fb23c070f80 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 -- 172.21.15.55:0/2036533428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 msgr2=0x7fb23c107320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 --2- 172.21.15.55:0/2036533428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009e50 tx=0x7fb22c009840 comp rx=0 tx=0).stop 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 -- 172.21.15.55:0/2036533428 shutdown_connections 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 --2- 172.21.15.55:0/2036533428 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c107320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 -- 172.21.15.55:0/2036533428 >> 172.21.15.55:0/2036533428 conn(0x7fb23c0fc970 msgr2=0x7fb23c0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 -- 172.21.15.55:0/2036533428 shutdown_connections 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 -- 172.21.15.55:0/2036533428 wait complete. 2023-12-19T19:05:44.127 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.353+0000 7fb241be2700 1 Processor -- start 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb241be2700 1 -- start start 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb241be2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb241be2700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c1138e0 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb23b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb23b7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58910/0 (socket says 172.21.15.55:58910) 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb23b7fe700 1 -- 172.21.15.55:0/1579814070 learned_addr learned my addr 172.21.15.55:0/1579814070 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb23b7fe700 1 -- 172.21.15.55:0/1579814070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009ae0 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.354+0000 7fb23b7fe700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000f80 tx=0x7fb22c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb22c01c620 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb22c01cbf0 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c113ae0 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb22c007b40 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c114780 con 0x7fb23c070f80 2023-12-19T19:05:44.128 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.355+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb22c019070 con 0x7fb23c070f80 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.356+0000 7fb223fff700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 0x7fb224038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.356+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb22c049f10 con 0x7fb23c070f80 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.356+0000 7fb23affd700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 0x7fb224038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.356+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c113f50 con 0x7fb23c070f80 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.356+0000 7fb23affd700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 0x7fb224038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb230009990 tx=0x7fb230006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.361+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb22c017080 con 0x7fb23c070f80 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.569+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb23c1149c0 con 0x7fb224036640 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.573+0000 7fb223fff700 1 -- 172.21.15.55:0/1579814070 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb23c1149c0 con 0x7fb224036640 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 msgr2=0x7fb224038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 0x7fb224038af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb230009990 tx=0x7fb230006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 msgr2=0x7fb23c1133a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000f80 tx=0x7fb22c009840 comp rx=0 tx=0).stop 2023-12-19T19:05:44.129 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 shutdown_connections 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb224036640 0x7fb224038af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 --2- 172.21.15.55:0/1579814070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb23c070f80 0x7fb23c1133a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.575+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 >> 172.21.15.55:0/1579814070 conn(0x7fb23c0fc970 msgr2=0x7fb23c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.576+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 shutdown_connections 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:42.576+0000 7fb241be2700 1 -- 172.21.15.55:0/1579814070 wait complete. 2023-12-19T19:05:44.130 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying mgr service with default placement... 2023-12-19T19:05:45.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:44 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:45.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:44 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:45.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:44 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:45.444 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb660eb700 1 Processor -- start 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb660eb700 1 -- start start 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb660eb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb660eb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb60103100 con 0x7fdb601027b0 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58914/0 (socket says 172.21.15.55:58914) 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb5f7fe700 1 -- 172.21.15.55:0/1969247656 learned_addr learned my addr 172.21.15.55:0/1969247656 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:45.445 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.554+0000 7fdb5f7fe700 1 -- 172.21.15.55:0/1969247656 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb60103240 con 0x7fdb601027b0 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb5f7fe700 1 --2- 172.21.15.55:0/1969247656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000bb80 tx=0x7fdb5000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66259beec60384 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb5effd700 1 -- 172.21.15.55:0/1969247656 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb5000dd00 con 0x7fdb601027b0 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb5effd700 1 -- 172.21.15.55:0/1969247656 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb5000de60 con 0x7fdb601027b0 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb5effd700 1 -- 172.21.15.55:0/1969247656 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb500188d0 con 0x7fdb601027b0 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb660eb700 1 -- 172.21.15.55:0/1969247656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 msgr2=0x7fdb60102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.555+0000 7fdb660eb700 1 --2- 172.21.15.55:0/1969247656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000bb80 tx=0x7fdb5000b4f0 comp rx=0 tx=0).stop 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- 172.21.15.55:0/1969247656 shutdown_connections 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 --2- 172.21.15.55:0/1969247656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb60102bc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:45.446 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- 172.21.15.55:0/1969247656 >> 172.21.15.55:0/1969247656 conn(0x7fdb600fc0b0 msgr2=0x7fdb600fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- 172.21.15.55:0/1969247656 shutdown_connections 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- 172.21.15.55:0/1969247656 wait complete. 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 Processor -- start 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- start start 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.556+0000 7fdb660eb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6010f470 con 0x7fdb601027b0 2023-12-19T19:05:45.447 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58916/0 (socket says 172.21.15.55:58916) 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5f7fe700 1 -- 172.21.15.55:0/605666443 learned_addr learned my addr 172.21.15.55:0/605666443 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5f7fe700 1 -- 172.21.15.55:0/605666443 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5000b790 con 0x7fdb601027b0 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5f7fe700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb500184f0 tx=0x7fdb5000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb50010b90 con 0x7fdb601027b0 2023-12-19T19:05:45.448 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb5001d050 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6010f670 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb50022590 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.557+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6010fa80 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.558+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6004bb80 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.558+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fdb50022850 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.559+0000 7fdb5cff9700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 0x7fdb4c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.559+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdb50014070 con 0x7fdb601027b0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.559+0000 7fdb57fff700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 0x7fdb4c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:45.449 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.559+0000 7fdb57fff700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 0x7fdb4c038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009990 tx=0x7fdb48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:45.450 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.564+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdb5007ea00 con 0x7fdb601027b0 2023-12-19T19:05:45.450 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.788+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb60108cd0 con 0x7fdb4c0365f0 2023-12-19T19:05:45.450 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.792+0000 7fdb5cff9700 1 -- 172.21.15.55:0/605666443 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdb60108cd0 con 0x7fdb4c0365f0 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 msgr2=0x7fdb4c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 0x7fdb4c038aa0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009990 tx=0x7fdb48006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 msgr2=0x7fdb6010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb500184f0 tx=0x7fdb5000b4f0 comp rx=0 tx=0).stop 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 shutdown_connections 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fdb4c0365f0 0x7fdb4c038aa0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 --2- 172.21.15.55:0/605666443 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb601027b0 0x7fdb6010ef30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:45.451 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 >> 172.21.15.55:0/605666443 conn(0x7fdb600fc0b0 msgr2=0x7fdb60105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:45.452 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 shutdown_connections 2023-12-19T19:05:45.452 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:44.794+0000 7fdb660eb700 1 -- 172.21.15.55:0/605666443 wait complete. 2023-12-19T19:05:45.452 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying crash service with default placement... 2023-12-19T19:05:46.074 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:45 smithi055 ceph-mon[80978]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:46.074 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:45 smithi055 ceph-mon[80978]: Saving service mgr spec with placement count:2 2023-12-19T19:05:46.944 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2023-12-19T19:05:46.944 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.183+0000 7f751d4c5700 1 Processor -- start 2023-12-19T19:05:46.944 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.183+0000 7f751d4c5700 1 -- start start 2023-12-19T19:05:46.944 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.183+0000 7f751d4c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.183+0000 7f751d4c5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75181053e0 con 0x7f7518071000 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7516ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7516ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58930/0 (socket says 172.21.15.55:58930) 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7516ffd700 1 -- 172.21.15.55:0/2462751314 learned_addr learned my addr 172.21.15.55:0/2462751314 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7516ffd700 1 -- 172.21.15.55:0/2462751314 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7518105520 con 0x7f7518071000 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7516ffd700 1 --2- 172.21.15.55:0/2462751314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36eb2faa398c1a server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7515ffb700 1 -- 172.21.15.55:0/2462751314 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f750000b860 con 0x7f7518071000 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7515ffb700 1 -- 172.21.15.55:0/2462751314 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f750000be70 con 0x7f7518071000 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.184+0000 7f7515ffb700 1 -- 172.21.15.55:0/2462751314 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7500010380 con 0x7f7518071000 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 -- 172.21.15.55:0/2462751314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 msgr2=0x7f7518104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 --2- 172.21.15.55:0/2462751314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f7500009250 comp rx=0 tx=0).stop 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 -- 172.21.15.55:0/2462751314 shutdown_connections 2023-12-19T19:05:46.945 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 --2- 172.21.15.55:0/2462751314 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518104ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 -- 172.21.15.55:0/2462751314 >> 172.21.15.55:0/2462751314 conn(0x7f75180fc970 msgr2=0x7f75180fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 -- 172.21.15.55:0/2462751314 shutdown_connections 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.185+0000 7f751d4c5700 1 -- 172.21.15.55:0/2462751314 wait complete. 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f751d4c5700 1 Processor -- start 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f751d4c5700 1 -- start start 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f751d4c5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f751d4c5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518100da0 con 0x7f7518071000 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f7516ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f7516ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58946/0 (socket says 172.21.15.55:58946) 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f7516ffd700 1 -- 172.21.15.55:0/169200581 learned_addr learned my addr 172.21.15.55:0/169200581 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f7516ffd700 1 -- 172.21.15.55:0/169200581 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000094f0 con 0x7f7518071000 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.186+0000 7f7516ffd700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7500003080 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.187+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7500010960 con 0x7f7518071000 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.187+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f750001a430 con 0x7f7518071000 2023-12-19T19:05:46.946 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.187+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7518100fa0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.187+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75000245b0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.187+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75181013b0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.188+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f7500010ac0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.188+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751810b180 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.188+0000 7f750f7fe700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 0x7f7504038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.188+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f750004a0c0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.188+0000 7f75167fc700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 0x7f7504038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.189+0000 7f75167fc700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 0x7f7504038af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7508009990 tx=0x7f7508006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.193+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f750002b4d0 con 0x7f7518071000 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.417+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f751802b150 con 0x7f7504036640 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.423+0000 7f750f7fe700 1 -- 172.21.15.55:0/169200581 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f751802b150 con 0x7f7504036640 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.425+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 msgr2=0x7f7504038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.425+0000 7f751d4c5700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 0x7f7504038af0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7508009990 tx=0x7f7508006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:46.947 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.425+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 msgr2=0x7f7518102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.425+0000 7f751d4c5700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7500003080 tx=0x7f7500009250 comp rx=0 tx=0).stop 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 shutdown_connections 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7504036640 0x7f7504038af0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 --2- 172.21.15.55:0/169200581 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7518071000 0x7f7518102920 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 >> 172.21.15.55:0/169200581 conn(0x7f75180fc970 msgr2=0x7f75180fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 shutdown_connections 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:46.426+0000 7f751d4c5700 1 -- 172.21.15.55:0/169200581 wait complete. 2023-12-19T19:05:46.948 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying prometheus service with default placement... 2023-12-19T19:05:47.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:47 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:47.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:47 smithi055 ceph-mon[80978]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:47.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:47 smithi055 ceph-mon[80978]: Saving service crash spec with placement * 2023-12-19T19:05:47.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:47 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:47.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:47 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8dd830700 1 Processor -- start 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8dd830700 1 -- start start 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8dd830700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8dd830700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8079130 con 0x7fa8d8078d20 2023-12-19T19:05:48.186 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8d6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8d6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58962/0 (socket says 172.21.15.55:58962) 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.390+0000 7fa8d6ffd700 1 -- 172.21.15.55:0/2157492546 learned_addr learned my addr 172.21.15.55:0/2157492546 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.391+0000 7fa8d6ffd700 1 -- 172.21.15.55:0/2157492546 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8077770 con 0x7fa8d8078d20 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.391+0000 7fa8d6ffd700 1 --2- 172.21.15.55:0/2157492546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80098e0 tx=0x7fa8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0d9b559c6caac9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.391+0000 7fa8d5ffb700 1 -- 172.21.15.55:0/2157492546 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c800b860 con 0x7fa8d8078d20 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.391+0000 7fa8d5ffb700 1 -- 172.21.15.55:0/2157492546 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8c800be70 con 0x7fa8d8078d20 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.391+0000 7fa8d5ffb700 1 -- 172.21.15.55:0/2157492546 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c8010380 con 0x7fa8d8078d20 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 -- 172.21.15.55:0/2157492546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 msgr2=0x7fa8d8077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2157492546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8c80098e0 tx=0x7fa8c8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 -- 172.21.15.55:0/2157492546 shutdown_connections 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2157492546 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8077230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:48.187 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 -- 172.21.15.55:0/2157492546 >> 172.21.15.55:0/2157492546 conn(0x7fa8d80fda10 msgr2=0x7fa8d80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 -- 172.21.15.55:0/2157492546 shutdown_connections 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.392+0000 7fa8dd830700 1 -- 172.21.15.55:0/2157492546 wait complete. 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8dd830700 1 Processor -- start 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8dd830700 1 -- start start 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8dd830700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8dd830700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8117cc0 con 0x7fa8d8078d20 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8d6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8d6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58972/0 (socket says 172.21.15.55:58972) 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.393+0000 7fa8d6ffd700 1 -- 172.21.15.55:0/2452630456 learned_addr learned my addr 172.21.15.55:0/2452630456 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8d6ffd700 1 -- 172.21.15.55:0/2452630456 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c80094f0 con 0x7fa8d8078d20 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8d6ffd700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8006fd0 tx=0x7fa8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c80105f0 con 0x7fa8d8078d20 2023-12-19T19:05:48.188 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8c8010c00 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d8117ec0 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8c801ad50 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.394+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d81182d0 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.395+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fa8c8025400 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.395+0000 7fa8bf7fe700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 0x7fa8c0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.395+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa8c8010d70 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.395+0000 7fa8d67fc700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 0x7fa8c0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.396+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d804bb80 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.396+0000 7fa8d67fc700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 0x7fa8c0038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009990 tx=0x7fa8cc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.401+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa8c8020070 con 0x7fa8d8078d20 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.620+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8d8110ce0 con 0x7fa8c0036640 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.624+0000 7fa8bf7fe700 1 -- 172.21.15.55:0/2452630456 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa8d8110ce0 con 0x7fa8c0036640 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 msgr2=0x7fa8c0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:48.189 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 0x7fa8c0038af0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009990 tx=0x7fa8cc006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 msgr2=0x7fa8d8117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8006fd0 tx=0x7fa8c8009250 comp rx=0 tx=0).stop 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 shutdown_connections 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fa8c0036640 0x7fa8c0038af0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 --2- 172.21.15.55:0/2452630456 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa8d8078d20 0x7fa8d8117780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 >> 172.21.15.55:0/2452630456 conn(0x7fa8d80fda10 msgr2=0x7fa8d80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 shutdown_connections 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:47.626+0000 7fa8dd830700 1 -- 172.21.15.55:0/2452630456 wait complete. 2023-12-19T19:05:48.190 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying grafana service with default placement... 2023-12-19T19:05:48.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:48 smithi055 ceph-mon[80978]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:48.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:48 smithi055 ceph-mon[80978]: Saving service prometheus spec with placement count:1 2023-12-19T19:05:48.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:48 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.063+0000 7f55b0b3a700 1 Processor -- start 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.063+0000 7f55b0b3a700 1 -- start start 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55b0b3a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55b0b3a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1053e0 con 0x7f55ac076c60 2023-12-19T19:05:49.897 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51388/0 (socket says 172.21.15.55:51388) 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55aa59c700 1 -- 172.21.15.55:0/3756830485 learned_addr learned my addr 172.21.15.55:0/3756830485 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55aa59c700 1 -- 172.21.15.55:0/3756830485 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac105520 con 0x7f55ac076c60 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.064+0000 7f55aa59c700 1 --2- 172.21.15.55:0/3756830485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55940098e0 tx=0x7f5594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedfd608973dff1e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55a959a700 1 -- 172.21.15.55:0/3756830485 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f559400b860 con 0x7f55ac076c60 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55a959a700 1 -- 172.21.15.55:0/3756830485 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f559400be70 con 0x7f55ac076c60 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55a959a700 1 -- 172.21.15.55:0/3756830485 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5594010380 con 0x7f55ac076c60 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 -- 172.21.15.55:0/3756830485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 msgr2=0x7f55ac104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/3756830485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55940098e0 tx=0x7f5594009250 comp rx=0 tx=0).stop 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 -- 172.21.15.55:0/3756830485 shutdown_connections 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/3756830485 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac104ea0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 -- 172.21.15.55:0/3756830485 >> 172.21.15.55:0/3756830485 conn(0x7f55ac0fc950 msgr2=0x7f55ac0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 -- 172.21.15.55:0/3756830485 shutdown_connections 2023-12-19T19:05:49.898 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.065+0000 7f55b0b3a700 1 -- 172.21.15.55:0/3756830485 wait complete. 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55b0b3a700 1 Processor -- start 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55b0b3a700 1 -- start start 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55b0b3a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55b0b3a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac1116d0 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55aa59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51390/0 (socket says 172.21.15.55:51390) 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.066+0000 7f55aa59c700 1 -- 172.21.15.55:0/1061919334 learned_addr learned my addr 172.21.15.55:0/1061919334 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55aa59c700 1 -- 172.21.15.55:0/1061919334 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55940094f0 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55aa59c700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5594006d40 tx=0x7f5594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5594010960 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f559401a430 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac1118d0 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55940245b0 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.067+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac111ce0 con 0x7f55ac076c60 2023-12-19T19:05:49.899 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.068+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f559402d3f0 con 0x7f55ac076c60 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.068+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac04bb80 con 0x7f55ac076c60 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.068+0000 7f55a2ffd700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 0x7f5598038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.068+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f559404a270 con 0x7f55ac076c60 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.068+0000 7f55a9d9b700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 0x7f5598038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.069+0000 7f55a9d9b700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 0x7f5598038aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f559c009990 tx=0x7f559c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.073+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5594010ca0 con 0x7f55ac076c60 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.283+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f55ac0008d0 con 0x7f55980365f0 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.288+0000 7f55a2ffd700 1 -- 172.21.15.55:0/1061919334 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f55ac0008d0 con 0x7f55980365f0 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 msgr2=0x7f5598038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 0x7f5598038aa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f559c009990 tx=0x7f559c006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 msgr2=0x7f55ac111190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5594006d40 tx=0x7f5594009250 comp rx=0 tx=0).stop 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 shutdown_connections 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f55980365f0 0x7f5598038aa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:49.900 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 --2- 172.21.15.55:0/1061919334 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f55ac076c60 0x7f55ac111190 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:49.901 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 >> 172.21.15.55:0/1061919334 conn(0x7f55ac0fc950 msgr2=0x7f55ac0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:49.901 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 shutdown_connections 2023-12-19T19:05:49.901 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:49.290+0000 7f55b0b3a700 1 -- 172.21.15.55:0/1061919334 wait complete. 2023-12-19T19:05:49.901 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying node-exporter service with default placement... 2023-12-19T19:05:50.528 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:50 smithi055 ceph-mon[80978]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:50.528 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:50 smithi055 ceph-mon[80978]: Saving service grafana spec with placement count:1 2023-12-19T19:05:50.529 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:50 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:51.028 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.309+0000 7f759f716700 1 Processor -- start 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.309+0000 7f759f716700 1 -- start start 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.309+0000 7f759f716700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.309+0000 7f759f716700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598104310 con 0x7f75981039c0 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f759d4b2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f759d4b2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51396/0 (socket says 172.21.15.55:51396) 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f759d4b2700 1 -- 172.21.15.55:0/178373486 learned_addr learned my addr 172.21.15.55:0/178373486 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f759d4b2700 1 -- 172.21.15.55:0/178373486 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598101fb0 con 0x7f75981039c0 2023-12-19T19:05:51.029 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f759d4b2700 1 --2- 172.21.15.55:0/178373486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f7588009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76c68d7ff00a65e server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f7597fff700 1 -- 172.21.15.55:0/178373486 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75880092e0 con 0x7f75981039c0 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f7597fff700 1 -- 172.21.15.55:0/178373486 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7588012430 con 0x7f75981039c0 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.310+0000 7f7597fff700 1 -- 172.21.15.55:0/178373486 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75880075c0 con 0x7f75981039c0 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- 172.21.15.55:0/178373486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 msgr2=0x7f7598103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 --2- 172.21.15.55:0/178373486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f7588009840 comp rx=0 tx=0).stop 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- 172.21.15.55:0/178373486 shutdown_connections 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 --2- 172.21.15.55:0/178373486 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598103dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- 172.21.15.55:0/178373486 >> 172.21.15.55:0/178373486 conn(0x7f75980fda40 msgr2=0x7f75980ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- 172.21.15.55:0/178373486 shutdown_connections 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- 172.21.15.55:0/178373486 wait complete. 2023-12-19T19:05:51.030 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 Processor -- start 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.311+0000 7f759f716700 1 -- start start 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759f716700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759f716700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598077190 con 0x7f75981039c0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759d4b2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759d4b2700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51400/0 (socket says 172.21.15.55:51400) 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759d4b2700 1 -- 172.21.15.55:0/1774786404 learned_addr learned my addr 172.21.15.55:0/1774786404 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759d4b2700 1 -- 172.21.15.55:0/1774786404 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588009ae0 con 0x7f75981039c0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759d4b2700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75880127d0 tx=0x7f7588009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75880092e0 con 0x7f75981039c0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7588007ce0 con 0x7f75981039c0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7598077390 con 0x7f75981039c0 2023-12-19T19:05:51.031 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.312+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f758801c910 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.313+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7598077800 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.313+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f7588019070 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.314+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759810b190 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.314+0000 7f7595ffb700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 0x7f7580038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.314+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7588049bf0 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.314+0000 7f759ccb1700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 0x7f7580038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.314+0000 7f759ccb1700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 0x7f7580038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f758c009990 tx=0x7f758c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.318+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7588019ad0 con 0x7f75981039c0 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.526+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f759802b180 con 0x7f7580036640 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.535+0000 7f7595ffb700 1 -- 172.21.15.55:0/1774786404 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f759802b180 con 0x7f7580036640 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 msgr2=0x7f7580038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 0x7f7580038af0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f758c009990 tx=0x7f758c006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:51.032 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 msgr2=0x7f7598078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75880127d0 tx=0x7f7588009840 comp rx=0 tx=0).stop 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 shutdown_connections 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f7580036640 0x7f7580038af0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 --2- 172.21.15.55:0/1774786404 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f75981039c0 0x7f7598078d10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 >> 172.21.15.55:0/1774786404 conn(0x7f75980fda40 msgr2=0x7f75980ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 shutdown_connections 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:50.537+0000 7f759f716700 1 -- 172.21.15.55:0/1774786404 wait complete. 2023-12-19T19:05:51.033 INFO:teuthology.orchestra.run.smithi055.stdout:Deploying alertmanager service with default placement... 2023-12-19T19:05:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:51 smithi055 ceph-mon[80978]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:51 smithi055 ceph-mon[80978]: Saving service node-exporter spec with placement * 2023-12-19T19:05:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:51 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.401+0000 7fd8ed86d700 1 Processor -- start 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.401+0000 7fd8ed86d700 1 -- start start 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.401+0000 7fd8ed86d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.401+0000 7fd8ed86d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e80790f0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51414/0 (socket says 172.21.15.55:51414) 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e6ffd700 1 -- 172.21.15.55:0/3401738179 learned_addr learned my addr 172.21.15.55:0/3401738179 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e6ffd700 1 -- 172.21.15.55:0/3401738179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8077730 con 0x7fd8e8078ce0 2023-12-19T19:05:52.160 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e6ffd700 1 --2- 172.21.15.55:0/3401738179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009e50 tx=0x7fd8e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0b09b73f2b2e0b server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e5ffb700 1 -- 172.21.15.55:0/3401738179 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e00092e0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e5ffb700 1 -- 172.21.15.55:0/3401738179 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8e0012430 con 0x7fd8e8078ce0 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.402+0000 7fd8e5ffb700 1 -- 172.21.15.55:0/3401738179 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e00075c0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 -- 172.21.15.55:0/3401738179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 msgr2=0x7fd8e80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/3401738179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009e50 tx=0x7fd8e0009840 comp rx=0 tx=0).stop 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 -- 172.21.15.55:0/3401738179 shutdown_connections 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/3401738179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e80771f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 -- 172.21.15.55:0/3401738179 >> 172.21.15.55:0/3401738179 conn(0x7fd8e80fda10 msgr2=0x7fd8e80ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 -- 172.21.15.55:0/3401738179 shutdown_connections 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 -- 172.21.15.55:0/3401738179 wait complete. 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.403+0000 7fd8ed86d700 1 Processor -- start 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8ed86d700 1 -- start start 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8ed86d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8ed86d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e81138e0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.161 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8e6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:52.162 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8e6ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51422/0 (socket says 172.21.15.55:51422) 2023-12-19T19:05:52.162 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8e6ffd700 1 -- 172.21.15.55:0/1453321293 learned_addr learned my addr 172.21.15.55:0/1453321293 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:52.162 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8e6ffd700 1 -- 172.21.15.55:0/1453321293 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e0009ae0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.162 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8e6ffd700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000bc50 tx=0x7fd8e0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e00092e0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8e0012900 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e8113ae0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e0025970 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.404+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e8113ef0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.405+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fd8e0019050 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.405+0000 7fd8df7fe700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 0x7fd8d4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.405+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd8e0049ed0 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.406+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e804bb80 con 0x7fd8e8078ce0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.406+0000 7fd8e67fc700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 0x7fd8d4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.406+0000 7fd8e67fc700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 0x7fd8d4038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009990 tx=0x7fd8d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:52.163 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.410+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd8e0020080 con 0x7fd8e8078ce0 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.617+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8e810d130 con 0x7fd8d4036640 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.621+0000 7fd8df7fe700 1 -- 172.21.15.55:0/1453321293 <== mgr.14120 v2:172.21.15.55:6800/2922737294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fd8e810d130 con 0x7fd8d4036640 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 msgr2=0x7fd8d4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 0x7fd8d4038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009990 tx=0x7fd8d0006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 msgr2=0x7fd8e81133a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000bc50 tx=0x7fd8e0009840 comp rx=0 tx=0).stop 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 shutdown_connections 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd8d4036640 0x7fd8d4038af0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 --2- 172.21.15.55:0/1453321293 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd8e8078ce0 0x7fd8e81133a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.623+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 >> 172.21.15.55:0/1453321293 conn(0x7fd8e80fda10 msgr2=0x7fd8e80ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.624+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 shutdown_connections 2023-12-19T19:05:52.164 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:51.624+0000 7fd8ed86d700 1 -- 172.21.15.55:0/1453321293 wait complete. 2023-12-19T19:05:52.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:52 smithi055 ceph-mon[80978]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:05:52.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:52 smithi055 ceph-mon[80978]: Saving service alertmanager spec with placement count:1 2023-12-19T19:05:52.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:52 smithi055 ceph-mon[80978]: from='mgr.14120 172.21.15.55:0/483842772' entity='mgr.smithi055.dwclvp' 2023-12-19T19:05:52.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:52 smithi055 ceph-mon[80978]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e9925d700 1 Processor -- start 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e9925d700 1 -- start start 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e9925d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e9925d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94105420 con 0x7f5e94076c20 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e92d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e92d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51428/0 (socket says 172.21.15.55:51428) 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.508+0000 7f5e92d9d700 1 -- 172.21.15.55:0/1298212427 learned_addr learned my addr 172.21.15.55:0/1298212427 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.509+0000 7f5e92d9d700 1 -- 172.21.15.55:0/1298212427 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94105560 con 0x7f5e94076c20 2023-12-19T19:05:53.299 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.509+0000 7f5e92d9d700 1 --2- 172.21.15.55:0/1298212427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e840098e0 tx=0x7f5e84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca477aa70c735d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.509+0000 7f5e91d9b700 1 -- 172.21.15.55:0/1298212427 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e8400b860 con 0x7f5e94076c20 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.509+0000 7f5e91d9b700 1 -- 172.21.15.55:0/1298212427 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e8400be70 con 0x7f5e94076c20 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.509+0000 7f5e91d9b700 1 -- 172.21.15.55:0/1298212427 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e84010380 con 0x7f5e94076c20 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 -- 172.21.15.55:0/1298212427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 msgr2=0x7f5e94104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 --2- 172.21.15.55:0/1298212427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5e840098e0 tx=0x7f5e84009250 comp rx=0 tx=0).stop 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 -- 172.21.15.55:0/1298212427 shutdown_connections 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 --2- 172.21.15.55:0/1298212427 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e94104ee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 -- 172.21.15.55:0/1298212427 >> 172.21.15.55:0/1298212427 conn(0x7f5e940fc950 msgr2=0x7f5e940fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 -- 172.21.15.55:0/1298212427 shutdown_connections 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 -- 172.21.15.55:0/1298212427 wait complete. 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.510+0000 7f5e9925d700 1 Processor -- start 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e9925d700 1 -- start start 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e9925d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:53.300 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e9925d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e94111700 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e92d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e92d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51436/0 (socket says 172.21.15.55:51436) 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e92d9d700 1 -- 172.21.15.55:0/3441249231 learned_addr learned my addr 172.21.15.55:0/3441249231 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e92d9d700 1 -- 172.21.15.55:0/3441249231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840094f0 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e92d9d700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e840094c0 tx=0x7f5e84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e84010850 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e84010e60 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.511+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e94111900 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.512+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e8401a430 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.512+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e94111d10 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.512+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5e8401a660 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.512+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9404bb80 con 0x7f5e94076c20 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.513+0000 7f5e7b7fe700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 0x7f5e7c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:53.301 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.513+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5e8404a4c0 con 0x7f5e94076c20 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.513+0000 7f5e9259c700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 0x7f5e7c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.513+0000 7f5e9259c700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 0x7f5e7c038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009990 tx=0x7f5e88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.517+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e84023580 con 0x7f5e94076c20 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.713+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5e941062e0 con 0x7f5e94076c20 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.720+0000 7f5e7b7fe700 1 -- 172.21.15.55:0/3441249231 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5e840233a0 con 0x7f5e94076c20 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 msgr2=0x7f5e7c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 0x7f5e7c038af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5e88009990 tx=0x7f5e88006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 msgr2=0x7f5e941111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5e840094c0 tx=0x7f5e84009250 comp rx=0 tx=0).stop 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 shutdown_connections 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f5e7c036640 0x7f5e7c038af0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 --2- 172.21.15.55:0/3441249231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5e94076c20 0x7f5e941111c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 >> 172.21.15.55:0/3441249231 conn(0x7f5e940fc950 msgr2=0x7f5e940fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 shutdown_connections 2023-12-19T19:05:53.302 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:52.723+0000 7f5e9925d700 1 -- 172.21.15.55:0/3441249231 wait complete. 2023-12-19T19:05:53.890 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:53 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3441249231' entity='client.admin' 2023-12-19T19:05:54.506 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2543bd4700 1 Processor -- start 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2543bd4700 1 -- start start 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2543bd4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2543bd4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c104310 con 0x7f253c1039c0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2541970700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2541970700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51450/0 (socket says 172.21.15.55:51450) 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.684+0000 7f2541970700 1 -- 172.21.15.55:0/2532938688 learned_addr learned my addr 172.21.15.55:0/2532938688 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.685+0000 7f2541970700 1 -- 172.21.15.55:0/2532938688 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c101fb0 con 0x7f253c1039c0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.685+0000 7f2541970700 1 --2- 172.21.15.55:0/2532938688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25380098e0 tx=0x7f2538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493264f2723a9747 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.685+0000 7f254096e700 1 -- 172.21.15.55:0/2532938688 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f253800b860 con 0x7f253c1039c0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.685+0000 7f254096e700 1 -- 172.21.15.55:0/2532938688 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f253800be70 con 0x7f253c1039c0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.685+0000 7f254096e700 1 -- 172.21.15.55:0/2532938688 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2538010380 con 0x7f253c1039c0 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- 172.21.15.55:0/2532938688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 msgr2=0x7f253c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 --2- 172.21.15.55:0/2532938688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f25380098e0 tx=0x7f2538009250 comp rx=0 tx=0).stop 2023-12-19T19:05:54.507 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- 172.21.15.55:0/2532938688 shutdown_connections 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 --2- 172.21.15.55:0/2532938688 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c103dd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- 172.21.15.55:0/2532938688 >> 172.21.15.55:0/2532938688 conn(0x7f253c0fda40 msgr2=0x7f253c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- 172.21.15.55:0/2532938688 shutdown_connections 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- 172.21.15.55:0/2532938688 wait complete. 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 Processor -- start 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.686+0000 7f2543bd4700 1 -- start start 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2543bd4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2543bd4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c077190 con 0x7f253c1039c0 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2541970700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2541970700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51464/0 (socket says 172.21.15.55:51464) 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2541970700 1 -- 172.21.15.55:0/3740621690 learned_addr learned my addr 172.21.15.55:0/3740621690 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2541970700 1 -- 172.21.15.55:0/3740621690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380094f0 con 0x7f253c1039c0 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2541970700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f253800bac0 tx=0x7f2538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:54.508 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25380109b0 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f253801a430 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25380245b0 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c077390 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.687+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c0777a0 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.688+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f253801a5a0 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.688+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c05f270 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.688+0000 7f25327fc700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 0x7f2528038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.688+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f253804a520 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.689+0000 7f254116f700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 0x7f2528038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.689+0000 7f254116f700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 0x7f2528038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f252c009990 tx=0x7f252c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.693+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2538010b10 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.888+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f253c102d30 con 0x7f253c1039c0 2023-12-19T19:05:54.509 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.891+0000 7f25327fc700 1 -- 172.21.15.55:0/3740621690 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2538018770 con 0x7f253c1039c0 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 msgr2=0x7f2528038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 0x7f2528038af0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f252c009990 tx=0x7f252c006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 msgr2=0x7f253c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f253800bac0 tx=0x7f2538009250 comp rx=0 tx=0).stop 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 shutdown_connections 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f2528036640 0x7f2528038af0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 --2- 172.21.15.55:0/3740621690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f253c1039c0 0x7f253c078d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 >> 172.21.15.55:0/3740621690 conn(0x7f253c0fda40 msgr2=0x7f253c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 shutdown_connections 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:53.893+0000 7f2543bd4700 1 -- 172.21.15.55:0/3740621690 wait complete. 2023-12-19T19:05:54.510 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling the dashboard module... 2023-12-19T19:05:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:54 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3740621690' entity='client.admin' 2023-12-19T19:05:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:54 smithi055 ceph-mon[80978]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:05:56.163 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:55 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/670278418' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-19T19:05:56.421 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb61933c700 1 Processor -- start 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb61933c700 1 -- start start 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb61933c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb61933c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6141053e0 con 0x7fb614076c60 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51474/0 (socket says 172.21.15.55:51474) 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb612ffd700 1 -- 172.21.15.55:0/1954669547 learned_addr learned my addr 172.21.15.55:0/1954669547 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.898+0000 7fb612ffd700 1 -- 172.21.15.55:0/1954669547 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614105520 con 0x7fb614076c60 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.899+0000 7fb612ffd700 1 --2- 172.21.15.55:0/1954669547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21bc11bafd9a855 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.902+0000 7fb611ffb700 1 -- 172.21.15.55:0/1954669547 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb60400b860 con 0x7fb614076c60 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.902+0000 7fb611ffb700 1 -- 172.21.15.55:0/1954669547 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb60400be70 con 0x7fb614076c60 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.902+0000 7fb611ffb700 1 -- 172.21.15.55:0/1954669547 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb604010380 con 0x7fb614076c60 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 -- 172.21.15.55:0/1954669547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 msgr2=0x7fb614104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:56.422 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 --2- 172.21.15.55:0/1954669547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb6040098e0 tx=0x7fb604009250 comp rx=0 tx=0).stop 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 -- 172.21.15.55:0/1954669547 shutdown_connections 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 --2- 172.21.15.55:0/1954669547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614104ea0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 -- 172.21.15.55:0/1954669547 >> 172.21.15.55:0/1954669547 conn(0x7fb6140fc950 msgr2=0x7fb6140fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 -- 172.21.15.55:0/1954669547 shutdown_connections 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.903+0000 7fb61933c700 1 -- 172.21.15.55:0/1954669547 wait complete. 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb61933c700 1 Processor -- start 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb61933c700 1 -- start start 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb61933c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb61933c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb614100da0 con 0x7fb614076c60 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb612ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51476/0 (socket says 172.21.15.55:51476) 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb612ffd700 1 -- 172.21.15.55:0/670278418 learned_addr learned my addr 172.21.15.55:0/670278418 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.904+0000 7fb612ffd700 1 -- 172.21.15.55:0/670278418 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6040094f0 con 0x7fb614076c60 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb612ffd700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb604006d40 tx=0x7fb604009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:56.423 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb604010960 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb60401a430 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb614100fa0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6040245b0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.905+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6141013b0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fb60402d3f0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb5fb7fe700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 0x7fb5fc038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61404bb80 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb60404a060 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb6127fc700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 0x7fb5fc038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.906+0000 7fb6127fc700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 0x7fb5fc038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb608009990 tx=0x7fb608006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:54.911+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb60402b6b0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.162+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb614106300 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.916+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fb60401a9f0 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.917+0000 7fb5fb7fe700 1 -- 172.21.15.55:0/670278418 <== mon.0 v2:172.21.15.55:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb60402c350 con 0x7fb614076c60 2023-12-19T19:05:56.424 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 msgr2=0x7fb5fc038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 0x7fb5fc038af0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb608009990 tx=0x7fb608006e40 comp rx=0 tx=0).stop 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 msgr2=0x7fb614102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb604006d40 tx=0x7fb604009250 comp rx=0 tx=0).stop 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 shutdown_connections 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fb5fc036640 0x7fb5fc038af0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 --2- 172.21.15.55:0/670278418 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb614076c60 0x7fb614102920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.920+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 >> 172.21.15.55:0/670278418 conn(0x7fb6140fc950 msgr2=0x7fb6140fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.921+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 shutdown_connections 2023-12-19T19:05:56.425 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:55.921+0000 7fb61933c700 1 -- 172.21.15.55:0/670278418 wait complete. 2023-12-19T19:05:57.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:56 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/670278418' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-19T19:05:57.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:56 smithi055 ceph-mon[80978]: mgrmap e9: smithi055.dwclvp(active, since 23s) 2023-12-19T19:05:57.720 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:05:57.720 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-12-19T19:05:57.720 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-19T19:05:57.720 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "active_name": "smithi055.dwclvp", 2023-12-19T19:05:57.720 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.862+0000 7f6623ed1700 1 Processor -- start 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6623ed1700 1 -- start start 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6623ed1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6623ed1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c104310 con 0x7f661c1039c0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6621c6d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6621c6d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51512/0 (socket says 172.21.15.55:51512) 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6621c6d700 1 -- 172.21.15.55:0/738547217 learned_addr learned my addr 172.21.15.55:0/738547217 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6621c6d700 1 -- 172.21.15.55:0/738547217 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c101fb0 con 0x7f661c1039c0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.863+0000 7f6621c6d700 1 --2- 172.21.15.55:0/738547217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66140098e0 tx=0x7f6614009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67079ffd2f14680f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.864+0000 7f6620c6b700 1 -- 172.21.15.55:0/738547217 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661400b860 con 0x7f661c1039c0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.864+0000 7f6620c6b700 1 -- 172.21.15.55:0/738547217 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f661400be70 con 0x7f661c1039c0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.864+0000 7f6620c6b700 1 -- 172.21.15.55:0/738547217 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6614010380 con 0x7f661c1039c0 2023-12-19T19:05:57.721 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.864+0000 7f6623ed1700 1 -- 172.21.15.55:0/738547217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 msgr2=0x7f661c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.864+0000 7f6623ed1700 1 --2- 172.21.15.55:0/738547217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66140098e0 tx=0x7f6614009250 comp rx=0 tx=0).stop 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 -- 172.21.15.55:0/738547217 shutdown_connections 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 --2- 172.21.15.55:0/738547217 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c103dd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 -- 172.21.15.55:0/738547217 >> 172.21.15.55:0/738547217 conn(0x7f661c0fda40 msgr2=0x7f661c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 -- 172.21.15.55:0/738547217 shutdown_connections 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 -- 172.21.15.55:0/738547217 wait complete. 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 Processor -- start 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.865+0000 7f6623ed1700 1 -- start start 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6623ed1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6623ed1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c111670 con 0x7f661c1039c0 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6621c6d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6621c6d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:51528/0 (socket says 172.21.15.55:51528) 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6621c6d700 1 -- 172.21.15.55:0/3319733947 learned_addr learned my addr 172.21.15.55:0/3319733947 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6621c6d700 1 -- 172.21.15.55:0/3319733947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66140094f0 con 0x7f661c1039c0 2023-12-19T19:05:57.722 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6621c6d700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6614003080 tx=0x7f6614009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6614010800 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6614010e10 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c111870 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.866+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661401a430 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.867+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c111c80 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.867+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c102d30 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.868+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f6614024790 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.868+0000 7f661a7fc700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 0x7f6604038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.868+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f661404a680 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.868+0000 7f662146c700 1 -- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 msgr2=0x7f6604038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.868+0000 7f662146c700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 0x7f6604038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:56.872+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6614022bc0 con 0x7f661c1039c0 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.069+0000 7f662146c700 1 -- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 msgr2=0x7f6604038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:05:57.723 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.069+0000 7f662146c700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 0x7f6604038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.124+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f661c10ad00 con 0x7f661c1039c0 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.125+0000 7f661a7fc700 1 -- 172.21.15.55:0/3319733947 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f661402ce60 con 0x7f661c1039c0 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 msgr2=0x7f6604038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 0x7f6604038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 msgr2=0x7f661c111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6614003080 tx=0x7f6614009250 comp rx=0 tx=0).stop 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 shutdown_connections 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7f6604036690 0x7f6604038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 --2- 172.21.15.55:0/3319733947 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f661c1039c0 0x7f661c111130 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 >> 172.21.15.55:0/3319733947 conn(0x7f661c0fda40 msgr2=0x7f661c0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 shutdown_connections 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:57.127+0000 7f6623ed1700 1 -- 172.21.15.55:0/3319733947 wait complete. 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for the mgr to restart... 2023-12-19T19:05:57.724 INFO:teuthology.orchestra.run.smithi055.stdout:Waiting for mgr epoch 9... 2023-12-19T19:05:58.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:05:57 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3319733947' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: Active manager daemon smithi055.dwclvp restarted 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: Activating manager daemon smithi055.dwclvp 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: osdmap e3: 0 total, 0 up, 0 in 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: mgrmap e10: smithi055.dwclvp(active, starting, since 0.0127012s) 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi055.dwclvp", "id": "smithi055.dwclvp"}]: dispatch 2023-12-19T19:06:03.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: Manager daemon smithi055.dwclvp is now available 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/mirror_snapshot_schedule"}]: dispatch 2023-12-19T19:06:03.800 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:03 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/trash_purge_schedule"}]: dispatch 2023-12-19T19:06:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout { 2023-12-19T19:06:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-12-19T19:06:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout } 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd1255fe700 1 Processor -- start 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd1255fe700 1 -- start start 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd1255fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd1255fe700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120104240 con 0x7fd1201038f0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd11effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd11effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52422/0 (socket says 172.21.15.55:52422) 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.105+0000 7fd11effd700 1 -- 172.21.15.55:0/3842957647 learned_addr learned my addr 172.21.15.55:0/3842957647 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.106+0000 7fd11effd700 1 -- 172.21.15.55:0/3842957647 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120104380 con 0x7fd1201038f0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.106+0000 7fd11effd700 1 --2- 172.21.15.55:0/3842957647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098e0 tx=0x7fd110009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4abd963c9ee1b7c server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.106+0000 7fd11dffb700 1 -- 172.21.15.55:0/3842957647 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11000b860 con 0x7fd1201038f0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.106+0000 7fd11dffb700 1 -- 172.21.15.55:0/3842957647 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd11000be70 con 0x7fd1201038f0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.106+0000 7fd11dffb700 1 -- 172.21.15.55:0/3842957647 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd110010380 con 0x7fd1201038f0 2023-12-19T19:06:04.512 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 -- 172.21.15.55:0/3842957647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 msgr2=0x7fd120103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 --2- 172.21.15.55:0/3842957647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1100098e0 tx=0x7fd110009250 comp rx=0 tx=0).stop 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 -- 172.21.15.55:0/3842957647 shutdown_connections 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 --2- 172.21.15.55:0/3842957647 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120103d00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 -- 172.21.15.55:0/3842957647 >> 172.21.15.55:0/3842957647 conn(0x7fd1200fc0b0 msgr2=0x7fd1200fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 -- 172.21.15.55:0/3842957647 shutdown_connections 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.107+0000 7fd1255fe700 1 -- 172.21.15.55:0/3842957647 wait complete. 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd1255fe700 1 Processor -- start 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd1255fe700 1 -- start start 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd1255fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd1255fe700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120111690 con 0x7fd1201038f0 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd11effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd11effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52424/0 (socket says 172.21.15.55:52424) 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd11effd700 1 -- 172.21.15.55:0/44840497 learned_addr learned my addr 172.21.15.55:0/44840497 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:04.513 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd11effd700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1100094f0 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.108+0000 7fd11effd700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd110011b60 tx=0x7fd110009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.109+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd110010850 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.109+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd110010e60 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.109+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd120111890 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.109+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11001ae20 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.109+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd120111ca0 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fd11002d3f0 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd1077fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd108039200 con 0x7fd108036640 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd11004a3c0 con 0x7fd1201038f0 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd11e7fc700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.110+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.311+0000 7fd11e7fc700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:06:04.514 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.311+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.712+0000 7fd11e7fc700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:58.712+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:59.513+0000 7fd11e7fc700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:05:59.513+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:01.114+0000 7fd11e7fc700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/2922737294 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:01.114+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:02.779+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7fd11002d740 con 0x7fd1201038f0 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:02.779+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 msgr2=0x7fd108038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:02.779+0000 7fd1077fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.776+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fd110017d90 con 0x7fd1201038f0 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.776+0000 7fd1077fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 0x7fd10803c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.776+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd108039200 con 0x7fd108039fb0 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.776+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 0x7fd10803c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.777+0000 7fd11e7fc700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 0x7fd10803c390 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd1140039d0 tx=0x7fd1140092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:04.515 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.777+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fd108039200 con 0x7fd108039fb0 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd12004bb80 con 0x7fd108039fb0 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1077fe700 1 -- 172.21.15.55:0/44840497 <== mgr.14162 v2:172.21.15.55:6800/797433438 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fd12004bb80 con 0x7fd108039fb0 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 msgr2=0x7fd10803c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 0x7fd10803c390 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd1140039d0 tx=0x7fd1140092f0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 msgr2=0x7fd120111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd110011b60 tx=0x7fd110009250 comp rx=0 tx=0).stop 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 shutdown_connections 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd1201038f0 0x7fd120111150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fd108039fb0 0x7fd10803c390 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 --2- 172.21.15.55:0/44840497 >> [v2:172.21.15.55:6800/2922737294,v1:172.21.15.55:6801/2922737294] conn(0x7fd108036640 0x7fd108038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 >> 172.21.15.55:0/44840497 conn(0x7fd1200fc0b0 msgr2=0x7fd1200fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 shutdown_connections 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:03.784+0000 7fd1255fe700 1 -- 172.21.15.55:0/44840497 wait complete. 2023-12-19T19:06:04.516 INFO:teuthology.orchestra.run.smithi055.stdout:mgr epoch 9 is available 2023-12-19T19:06:04.517 INFO:teuthology.orchestra.run.smithi055.stdout:Generating a dashboard self-signed certificate... 2023-12-19T19:06:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: [19/Dec/2023:19:06:03] ENGINE Bus STARTING 2023-12-19T19:06:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-19T19:06:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: mgrmap e11: smithi055.dwclvp(active, since 1.01089s) 2023-12-19T19:06:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-19T19:06:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: [19/Dec/2023:19:06:03] ENGINE Serving on https://172.21.15.55:7150 2023-12-19T19:06:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: [19/Dec/2023:19:06:03] ENGINE Bus STARTED 2023-12-19T19:06:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:04 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:05.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-12-19T19:06:05.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa925646700 1 Processor -- start 2023-12-19T19:06:05.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa925646700 1 -- start start 2023-12-19T19:06:05.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa925646700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:05.836 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa925646700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920105420 con 0x7fa920070fc0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa91effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa91effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52480/0 (socket says 172.21.15.55:52480) 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.954+0000 7fa91effd700 1 -- 172.21.15.55:0/3510512375 learned_addr learned my addr 172.21.15.55:0/3510512375 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.955+0000 7fa91effd700 1 -- 172.21.15.55:0/3510512375 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920105560 con 0x7fa920070fc0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.955+0000 7fa91effd700 1 --2- 172.21.15.55:0/3510512375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa910009e50 tx=0x7fa910009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab04a760d119dca server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.955+0000 7fa91dffb700 1 -- 172.21.15.55:0/3510512375 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9100092e0 con 0x7fa920070fc0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.955+0000 7fa91dffb700 1 -- 172.21.15.55:0/3510512375 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa910012430 con 0x7fa920070fc0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.955+0000 7fa91dffb700 1 -- 172.21.15.55:0/3510512375 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9100075c0 con 0x7fa920070fc0 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 -- 172.21.15.55:0/3510512375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 msgr2=0x7fa920104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 --2- 172.21.15.55:0/3510512375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa910009e50 tx=0x7fa910009840 comp rx=0 tx=0).stop 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 -- 172.21.15.55:0/3510512375 shutdown_connections 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 --2- 172.21.15.55:0/3510512375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa920104ee0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 -- 172.21.15.55:0/3510512375 >> 172.21.15.55:0/3510512375 conn(0x7fa9200fc970 msgr2=0x7fa9200fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:05.837 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 -- 172.21.15.55:0/3510512375 shutdown_connections 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.956+0000 7fa925646700 1 -- 172.21.15.55:0/3510512375 wait complete. 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa925646700 1 Processor -- start 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa925646700 1 -- start start 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa925646700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa925646700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa920111730 con 0x7fa920070fc0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa91effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa91effd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52494/0 (socket says 172.21.15.55:52494) 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.957+0000 7fa91effd700 1 -- 172.21.15.55:0/1868026800 learned_addr learned my addr 172.21.15.55:0/1868026800 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa91effd700 1 -- 172.21.15.55:0/1868026800 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa910009ae0 con 0x7fa920070fc0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa91effd700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa910009fd0 tx=0x7fa910009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9100092e0 con 0x7fa920070fc0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa910012920 con 0x7fa920070fc0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa920111930 con 0x7fa920070fc0 2023-12-19T19:06:05.838 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa920111d40 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.958+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9100258e0 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.959+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fa910019050 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.959+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92004bb80 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.959+0000 7fa9177fe700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 0x7fa90c03cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.959+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa91004adc0 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.959+0000 7fa91e7fc700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 0x7fa90c03cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.960+0000 7fa91e7fc700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 0x7fa90c03cdd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa908009990 tx=0x7fa908006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:04.964+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa91002e3f0 con 0x7fa920070fc0 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.188+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa92010aff0 con 0x7fa90c03a920 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.269+0000 7fa9177fe700 1 -- 172.21.15.55:0/1868026800 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa92010aff0 con 0x7fa90c03a920 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 msgr2=0x7fa90c03cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 0x7fa90c03cdd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa908009990 tx=0x7fa908006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:05.839 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 msgr2=0x7fa9201111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa910009fd0 tx=0x7fa910009840 comp rx=0 tx=0).stop 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 shutdown_connections 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa90c03a920 0x7fa90c03cdd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 --2- 172.21.15.55:0/1868026800 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa920070fc0 0x7fa9201111f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 >> 172.21.15.55:0/1868026800 conn(0x7fa9200fc970 msgr2=0x7fa9200fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 shutdown_connections 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:05.272+0000 7fa925646700 1 -- 172.21.15.55:0/1868026800 wait complete. 2023-12-19T19:06:05.840 INFO:teuthology.orchestra.run.smithi055.stdout:Creating initial admin user... 2023-12-19T19:06:06.526 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:06 smithi055 ceph-mon[80978]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:06:06.526 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:06 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:06.526 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:06 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:06.526 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:06 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:06.526 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:06 smithi055 ceph-mon[80978]: mgrmap e12: smithi055.dwclvp(active, since 2s) 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$gpVe3tSpzHRjP4zu0cq9yOF8K629e8vgPMBQ9PjOTuuwwB8/pxZrC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703012766, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.209+0000 7f6be0ceb700 1 Processor -- start 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.209+0000 7f6be0ceb700 1 -- start start 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6be0ceb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6be0ceb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc103100 con 0x7f6bdc1027b0 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6bda59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6bda59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52504/0 (socket says 172.21.15.55:52504) 2023-12-19T19:06:07.339 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6bda59c700 1 -- 172.21.15.55:0/954845132 learned_addr learned my addr 172.21.15.55:0/954845132 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6bda59c700 1 -- 172.21.15.55:0/954845132 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc103240 con 0x7f6bdc1027b0 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.210+0000 7f6bda59c700 1 --2- 172.21.15.55:0/954845132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00bb80 tx=0x7f6bcc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3b2ff768a84419 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.211+0000 7f6bd959a700 1 -- 172.21.15.55:0/954845132 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bcc00dd00 con 0x7f6bdc1027b0 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.211+0000 7f6bd959a700 1 -- 172.21.15.55:0/954845132 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bcc00de60 con 0x7f6bdc1027b0 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.211+0000 7f6bd959a700 1 -- 172.21.15.55:0/954845132 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bcc0188d0 con 0x7f6bdc1027b0 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.211+0000 7f6be0ceb700 1 -- 172.21.15.55:0/954845132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 msgr2=0x7f6bdc102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.211+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/954845132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00bb80 tx=0x7f6bcc00b4f0 comp rx=0 tx=0).stop 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 -- 172.21.15.55:0/954845132 shutdown_connections 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/954845132 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc102bc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 -- 172.21.15.55:0/954845132 >> 172.21.15.55:0/954845132 conn(0x7f6bdc0fc0b0 msgr2=0x7f6bdc0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 -- 172.21.15.55:0/954845132 shutdown_connections 2023-12-19T19:06:07.340 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 -- 172.21.15.55:0/954845132 wait complete. 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 Processor -- start 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.212+0000 7f6be0ceb700 1 -- start start 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6be0ceb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6be0ceb700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc10f470 con 0x7f6bdc1027b0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bda59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bda59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52506/0 (socket says 172.21.15.55:52506) 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bda59c700 1 -- 172.21.15.55:0/214766381 learned_addr learned my addr 172.21.15.55:0/214766381 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bda59c700 1 -- 172.21.15.55:0/214766381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc00b790 con 0x7f6bdc1027b0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bda59c700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0184f0 tx=0x7f6bcc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bcc010bd0 con 0x7f6bdc1027b0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bcc01d050 con 0x7f6bdc1027b0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc10f670 con 0x7f6bdc1027b0 2023-12-19T19:06:07.341 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bcc022590 con 0x7f6bdc1027b0 2023-12-19T19:06:07.342 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.213+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc10fa80 con 0x7f6bdc1027b0 2023-12-19T19:06:07.342 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.214+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f6bcc022850 con 0x7f6bdc1027b0 2023-12-19T19:06:07.342 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.215+0000 7f6bcaffd700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 0x7f6bc4038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:07.342 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.215+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6bcc014070 con 0x7f6bdc1027b0 2023-12-19T19:06:07.342 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.215+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc04bb80 con 0x7f6bdc1027b0 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.215+0000 7f6bd9d9b700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 0x7f6bc4038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.215+0000 7f6bd9d9b700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 0x7f6bc4038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009990 tx=0x7f6bd0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.220+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6bcc022ba0 con 0x7f6bdc1027b0 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.449+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6bdc05f090 con 0x7f6bc4036640 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.696+0000 7f6bcaffd700 1 -- 172.21.15.55:0/214766381 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f6bdc05f090 con 0x7f6bc4036640 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 msgr2=0x7f6bc4038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 0x7f6bc4038af0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0009990 tx=0x7f6bd0006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 msgr2=0x7f6bdc10ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0184f0 tx=0x7f6bcc00b4f0 comp rx=0 tx=0).stop 2023-12-19T19:06:07.343 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 shutdown_connections 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6bc4036640 0x7f6bc4038af0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 --2- 172.21.15.55:0/214766381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6bdc1027b0 0x7f6bdc10ef30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 >> 172.21.15.55:0/214766381 conn(0x7f6bdc0fc0b0 msgr2=0x7f6bdc105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 shutdown_connections 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:06.698+0000 7f6be0ceb700 1 -- 172.21.15.55:0/214766381 wait complete. 2023-12-19T19:06:07.344 INFO:teuthology.orchestra.run.smithi055.stdout:Fetching dashboard port number... 2023-12-19T19:06:07.946 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:07 smithi055 ceph-mon[80978]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:06:07.946 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:07 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stdout 8443 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.733+0000 7f007d1c3700 1 Processor -- start 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.733+0000 7f007d1c3700 1 -- start start 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.733+0000 7f007d1c3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.733+0000 7f007d1c3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078105440 con 0x7f0078070fa0 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60066/0 (socket says 172.21.15.55:60066) 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0076d9d700 1 -- 172.21.15.55:0/1972110563 learned_addr learned my addr 172.21.15.55:0/1972110563 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0076d9d700 1 -- 172.21.15.55:0/1972110563 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0078105580 con 0x7f0078070fa0 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0076d9d700 1 --2- 172.21.15.55:0/1972110563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098e0 tx=0x7f006c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c184bd8edda1e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:08.639 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0075d9b700 1 -- 172.21.15.55:0/1972110563 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c00b860 con 0x7f0078070fa0 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0075d9b700 1 -- 172.21.15.55:0/1972110563 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f006c00be70 con 0x7f0078070fa0 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.734+0000 7f0075d9b700 1 -- 172.21.15.55:0/1972110563 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c010380 con 0x7f0078070fa0 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 -- 172.21.15.55:0/1972110563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 msgr2=0x7f0078104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 --2- 172.21.15.55:0/1972110563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f006c0098e0 tx=0x7f006c009250 comp rx=0 tx=0).stop 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 -- 172.21.15.55:0/1972110563 shutdown_connections 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 --2- 172.21.15.55:0/1972110563 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f0078104f00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 -- 172.21.15.55:0/1972110563 >> 172.21.15.55:0/1972110563 conn(0x7f00780fc970 msgr2=0x7f00780fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 -- 172.21.15.55:0/1972110563 shutdown_connections 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.735+0000 7f007d1c3700 1 -- 172.21.15.55:0/1972110563 wait complete. 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f007d1c3700 1 Processor -- start 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f007d1c3700 1 -- start start 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f007d1c3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f007d1c3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078111700 con 0x7f0078070fa0 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f0076d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60080/0 (socket says 172.21.15.55:60080) 2023-12-19T19:06:08.640 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f0076d9d700 1 -- 172.21.15.55:0/2299109146 learned_addr learned my addr 172.21.15.55:0/2299109146 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f0076d9d700 1 -- 172.21.15.55:0/2299109146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0094f0 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.736+0000 7f0076d9d700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f006c0094c0 tx=0x7f006c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.737+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c010850 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.737+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f006c010e60 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.737+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0078111900 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.737+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f006c01a430 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.737+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0078111d10 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f006c01a660 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007804bb80 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f005f7fe700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 0x7f0060038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f006c04a4a0 con 0x7f0078070fa0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f007659c700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 0x7f0060038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.738+0000 7f007659c700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 0x7f0060038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0068009990 tx=0x7f0068006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:08.641 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.743+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f006c0247b0 con 0x7f0078070fa0 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.943+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0078106300 con 0x7f0078070fa0 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.944+0000 7f005f7fe700 1 -- 172.21.15.55:0/2299109146 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f006c022340 con 0x7f0078070fa0 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 msgr2=0x7f0060038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 0x7f0060038b40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0068009990 tx=0x7f0068006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 msgr2=0x7f00781111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f006c0094c0 tx=0x7f006c009250 comp rx=0 tx=0).stop 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 shutdown_connections 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0060036690 0x7f0060038b40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 --2- 172.21.15.55:0/2299109146 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0078070fa0 0x7f00781111c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.946+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 >> 172.21.15.55:0/2299109146 conn(0x7f00780fc970 msgr2=0x7f00780fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.947+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 shutdown_connections 2023-12-19T19:06:08.642 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:07.947+0000 7f007d1c3700 1 -- 172.21.15.55:0/2299109146 wait complete. 2023-12-19T19:06:08.646 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-19T19:06:08.646 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout disabled 2023-12-19T19:06:08.653 INFO:teuthology.orchestra.run.smithi055.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-19T19:06:08.653 INFO:teuthology.orchestra.run.smithi055.stdout:systemctl: stdout inactive 2023-12-19T19:06:08.653 INFO:teuthology.orchestra.run.smithi055.stdout:firewalld.service is not enabled 2023-12-19T19:06:08.653 INFO:teuthology.orchestra.run.smithi055.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-19T19:06:08.655 INFO:teuthology.orchestra.run.smithi055.stdout:Ceph Dashboard is now available at: 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout: URL: https://smithi055.front.sepia.ceph.com:8443/ 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout: User: admin 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout: Password: 0tbw0q0kg3 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout:Saving cluster configuration to /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config directory 2023-12-19T19:06:08.656 INFO:teuthology.orchestra.run.smithi055.stdout:Enabling autotune for osd_memory_target 2023-12-19T19:06:08.951 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:08 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2299109146' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.051+0000 7f35c22c7700 1 Processor -- start 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35c22c7700 1 -- start start 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35c22c7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35c22c7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc105460 con 0x7f35bc076be0 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35bbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35bbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60082/0 (socket says 172.21.15.55:60082) 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35bbfff700 1 -- 172.21.15.55:0/1467136332 learned_addr learned my addr 172.21.15.55:0/1467136332 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35bbfff700 1 -- 172.21.15.55:0/1467136332 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc1055a0 con 0x7f35bc076be0 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.052+0000 7f35bbfff700 1 --2- 172.21.15.55:0/1467136332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0098e0 tx=0x7f35ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b81223ca7da9181 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:09.750 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35baffd700 1 -- 172.21.15.55:0/1467136332 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac00b860 con 0x7f35bc076be0 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35baffd700 1 -- 172.21.15.55:0/1467136332 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35ac00be70 con 0x7f35bc076be0 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35baffd700 1 -- 172.21.15.55:0/1467136332 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac010380 con 0x7f35bc076be0 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 -- 172.21.15.55:0/1467136332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 msgr2=0x7f35bc104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 --2- 172.21.15.55:0/1467136332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0098e0 tx=0x7f35ac009250 comp rx=0 tx=0).stop 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 -- 172.21.15.55:0/1467136332 shutdown_connections 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 --2- 172.21.15.55:0/1467136332 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc104f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 -- 172.21.15.55:0/1467136332 >> 172.21.15.55:0/1467136332 conn(0x7f35bc0fc950 msgr2=0x7f35bc0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 -- 172.21.15.55:0/1467136332 shutdown_connections 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.053+0000 7f35c22c7700 1 -- 172.21.15.55:0/1467136332 wait complete. 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35c22c7700 1 Processor -- start 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35c22c7700 1 -- start start 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35c22c7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35c22c7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc100e20 con 0x7f35bc076be0 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35bbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:09.751 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35bbfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60084/0 (socket says 172.21.15.55:60084) 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.054+0000 7f35bbfff700 1 -- 172.21.15.55:0/668441460 learned_addr learned my addr 172.21.15.55:0/668441460 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35bbfff700 1 -- 172.21.15.55:0/668441460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac0094f0 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35bbfff700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35ac006d40 tx=0x7f35ac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac010960 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35ac01a430 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc101020 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac0245b0 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.055+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc101430 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.056+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f35ac02d3f0 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.056+0000 7f35b8ff9700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 0x7f35a8038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.056+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f35ac04a330 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.056+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc10b180 con 0x7f35bc076be0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.056+0000 7f35bb7fe700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 0x7f35a8038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.057+0000 7f35bb7fe700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 0x7f35a8038b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009990 tx=0x7f35a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:09.752 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.061+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f35ac02b8e0 con 0x7f35bc076be0 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.256+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f35bc04bb80 con 0x7f35bc076be0 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.257+0000 7f35b8ff9700 1 -- 172.21.15.55:0/668441460 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f35ac02b2f0 con 0x7f35bc076be0 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 msgr2=0x7f35a8038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 0x7f35a8038b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009990 tx=0x7f35a4006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 msgr2=0x7f35bc1029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f35ac006d40 tx=0x7f35ac009250 comp rx=0 tx=0).stop 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 shutdown_connections 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f35a8036690 0x7f35a8038b40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 --2- 172.21.15.55:0/668441460 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f35bc076be0 0x7f35bc1029a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.258+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 >> 172.21.15.55:0/668441460 conn(0x7f35bc0fc950 msgr2=0x7f35bc0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.259+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 shutdown_connections 2023-12-19T19:06:09.753 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:09.259+0000 7f35c22c7700 1 -- 172.21.15.55:0/668441460 wait complete. 2023-12-19T19:06:10.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:10 smithi055 ceph-mon[80978]: mgrmap e13: smithi055.dwclvp(active, since 6s) 2023-12-19T19:06:10.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:10 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/19648476' entity='client.admin' 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa58606f700 1 Processor -- start 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa58606f700 1 -- start start 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa58606f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa58606f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5801053e0 con 0x7fa580071000 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60096/0 (socket says 172.21.15.55:60096) 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.143+0000 7fa57f7fe700 1 -- 172.21.15.55:0/3866919920 learned_addr learned my addr 172.21.15.55:0/3866919920 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:11.069 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.144+0000 7fa57f7fe700 1 -- 172.21.15.55:0/3866919920 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580105520 con 0x7fa580071000 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.144+0000 7fa57f7fe700 1 --2- 172.21.15.55:0/3866919920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098e0 tx=0x7fa578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fd0700bfbc001f server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.144+0000 7fa57e7fc700 1 -- 172.21.15.55:0/3866919920 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57800b860 con 0x7fa580071000 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.144+0000 7fa57e7fc700 1 -- 172.21.15.55:0/3866919920 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa57800be70 con 0x7fa580071000 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.144+0000 7fa57e7fc700 1 -- 172.21.15.55:0/3866919920 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa578010380 con 0x7fa580071000 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- 172.21.15.55:0/3866919920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 msgr2=0x7fa580104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 --2- 172.21.15.55:0/3866919920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa5780098e0 tx=0x7fa578009250 comp rx=0 tx=0).stop 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- 172.21.15.55:0/3866919920 shutdown_connections 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 --2- 172.21.15.55:0/3866919920 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa580104ea0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- 172.21.15.55:0/3866919920 >> 172.21.15.55:0/3866919920 conn(0x7fa5800fc970 msgr2=0x7fa5800fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- 172.21.15.55:0/3866919920 shutdown_connections 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- 172.21.15.55:0/3866919920 wait complete. 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 Processor -- start 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.145+0000 7fa58606f700 1 -- start start 2023-12-19T19:06:11.070 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa58606f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa58606f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa580111700 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa57f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60102/0 (socket says 172.21.15.55:60102) 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa57f7fe700 1 -- 172.21.15.55:0/19648476 learned_addr learned my addr 172.21.15.55:0/19648476 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa57f7fe700 1 -- 172.21.15.55:0/19648476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780094f0 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa57f7fe700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5780094c0 tx=0x7fa578009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa578010850 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa578010e60 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa580111900 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.146+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57801a430 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.147+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580111d10 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.147+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58004bb80 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.148+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fa5780247b0 con 0x7fa580071000 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.148+0000 7fa577fff700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 0x7fa56c038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:11.071 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.148+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa57804a610 con 0x7fa580071000 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.148+0000 7fa57effd700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 0x7fa56c038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.148+0000 7fa57effd700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 0x7fa56c038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa568009990 tx=0x7fa568006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.152+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa578022520 con 0x7fa580071000 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.466+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa580106300 con 0x7fa580071000 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.471+0000 7fa577fff700 1 -- 172.21.15.55:0/19648476 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa578022340 con 0x7fa580071000 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 msgr2=0x7fa56c038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 0x7fa56c038b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa568009990 tx=0x7fa568006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 msgr2=0x7fa5801111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5780094c0 tx=0x7fa578009250 comp rx=0 tx=0).stop 2023-12-19T19:06:11.072 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 shutdown_connections 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa56c0366e0 0x7fa56c038b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 --2- 172.21.15.55:0/19648476 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa580071000 0x7fa5801111c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 >> 172.21.15.55:0/19648476 conn(0x7fa5800fc970 msgr2=0x7fa5800fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 shutdown_connections 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr 2023-12-19T19:06:10.473+0000 7fa58606f700 1 -- 172.21.15.55:0/19648476 wait complete. 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:Or, if you are only running a single cluster on this host: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.073 INFO:teuthology.orchestra.run.smithi055.stdout:Please consider enabling telemetry to help improve Ceph: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: ceph telemetry on 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout:For more information see: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:11.074 INFO:teuthology.orchestra.run.smithi055.stdout:Bootstrap complete. 2023-12-19T19:06:11.111 INFO:tasks.cephadm:Fetching config... 2023-12-19T19:06:11.111 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:06:11.111 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-19T19:06:11.167 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-19T19:06:11.167 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:06:11.167 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-19T19:06:11.222 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-19T19:06:11.222 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:06:11.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/keyring of=/dev/stdout 2023-12-19T19:06:11.293 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-19T19:06:11.293 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:06:11.293 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-19T19:06:11.348 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-19T19:06:11.348 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZuGUl201Ipp4fIlxXhtPA4TLZy2TdN6TqKMc9rmWwmshCnBqdPRdT1FTJp3DpnwxhIVUpYHF946n30FZtd3T0cTtxfpZzUVoMQGjd6i3TTyz4xwdqncwuGsw5q+SSsf6bq3OmLGLkzljbyCT8YcPv6rfeO63W7FyA186KCz62w2dQTJVb+zhYnEFjkUA0iJY7mtN8lPsJjjDnm0FtTUcCkVu/PIKM4Sgh+qHyJHAPwh+hDW40HoPSDpdOIkw3Ec++d78m7jwo1QTTX3CaCoSK/+0AS2W6InNoTQej4HiXk5/ktgZULauH6Vsn9wVxwF4uOl3FY5t+bVHLSBRn0MWEjfzu2sWaO2vQkEkQhb7xKg9i3zUG9KBMg7XlD19payeUpoMZpWoFzNeDQ+ir1bVDFV1gKcOlNWlKbYInDTEjEmjfxmSGsf2fZoKl33dUmSTOslQQKfjC/EbmWa4O0OX/90Xb66+mfWlUTmp22+5+9lGNbOA3gfFsLmCwkvNteM= ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-19T19:06:11.436 INFO:teuthology.orchestra.run.smithi055.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZuGUl201Ipp4fIlxXhtPA4TLZy2TdN6TqKMc9rmWwmshCnBqdPRdT1FTJp3DpnwxhIVUpYHF946n30FZtd3T0cTtxfpZzUVoMQGjd6i3TTyz4xwdqncwuGsw5q+SSsf6bq3OmLGLkzljbyCT8YcPv6rfeO63W7FyA186KCz62w2dQTJVb+zhYnEFjkUA0iJY7mtN8lPsJjjDnm0FtTUcCkVu/PIKM4Sgh+qHyJHAPwh+hDW40HoPSDpdOIkw3Ec++d78m7jwo1QTTX3CaCoSK/+0AS2W6InNoTQej4HiXk5/ktgZULauH6Vsn9wVxwF4uOl3FY5t+bVHLSBRn0MWEjfzu2sWaO2vQkEkQhb7xKg9i3zUG9KBMg7XlD19payeUpoMZpWoFzNeDQ+ir1bVDFV1gKcOlNWlKbYInDTEjEmjfxmSGsf2fZoKl33dUmSTOslQQKfjC/EbmWa4O0OX/90Xb66+mfWlUTmp22+5+9lGNbOA3gfFsLmCwkvNteM= ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:06:11.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZuGUl201Ipp4fIlxXhtPA4TLZy2TdN6TqKMc9rmWwmshCnBqdPRdT1FTJp3DpnwxhIVUpYHF946n30FZtd3T0cTtxfpZzUVoMQGjd6i3TTyz4xwdqncwuGsw5q+SSsf6bq3OmLGLkzljbyCT8YcPv6rfeO63W7FyA186KCz62w2dQTJVb+zhYnEFjkUA0iJY7mtN8lPsJjjDnm0FtTUcCkVu/PIKM4Sgh+qHyJHAPwh+hDW40HoPSDpdOIkw3Ec++d78m7jwo1QTTX3CaCoSK/+0AS2W6InNoTQej4HiXk5/ktgZULauH6Vsn9wVxwF4uOl3FY5t+bVHLSBRn0MWEjfzu2sWaO2vQkEkQhb7xKg9i3zUG9KBMg7XlD19payeUpoMZpWoFzNeDQ+ir1bVDFV1gKcOlNWlKbYInDTEjEmjfxmSGsf2fZoKl33dUmSTOslQQKfjC/EbmWa4O0OX/90Xb66+mfWlUTmp22+5+9lGNbOA3gfFsLmCwkvNteM= ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-19T19:06:11.499 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwZuGUl201Ipp4fIlxXhtPA4TLZy2TdN6TqKMc9rmWwmshCnBqdPRdT1FTJp3DpnwxhIVUpYHF946n30FZtd3T0cTtxfpZzUVoMQGjd6i3TTyz4xwdqncwuGsw5q+SSsf6bq3OmLGLkzljbyCT8YcPv6rfeO63W7FyA186KCz62w2dQTJVb+zhYnEFjkUA0iJY7mtN8lPsJjjDnm0FtTUcCkVu/PIKM4Sgh+qHyJHAPwh+hDW40HoPSDpdOIkw3Ec++d78m7jwo1QTTX3CaCoSK/+0AS2W6InNoTQej4HiXk5/ktgZULauH6Vsn9wVxwF4uOl3FY5t+bVHLSBRn0MWEjfzu2sWaO2vQkEkQhb7xKg9i3zUG9KBMg7XlD19payeUpoMZpWoFzNeDQ+ir1bVDFV1gKcOlNWlKbYInDTEjEmjfxmSGsf2fZoKl33dUmSTOslQQKfjC/EbmWa4O0OX/90Xb66+mfWlUTmp22+5+9lGNbOA3gfFsLmCwkvNteM= ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:06:11.514 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-19T19:06:11.775 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 -- 172.21.15.55:0/3839128233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 msgr2=0x7f6d3c06ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 --2- 172.21.15.55:0/3839128233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c06ffd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6d240048c0 tx=0x7f6d24009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 -- 172.21.15.55:0/3839128233 shutdown_connections 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 --2- 172.21.15.55:0/3839128233 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c06ffd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 -- 172.21.15.55:0/3839128233 >> 172.21.15.55:0/3839128233 conn(0x7f6d3c0f7f20 msgr2=0x7f6d3c0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 -- 172.21.15.55:0/3839128233 shutdown_connections 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.196+0000 7f6d42935700 1 -- 172.21.15.55:0/3839128233 wait complete. 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d42935700 1 Processor -- start 2023-12-19T19:06:14.198 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d42935700 1 -- start start 2023-12-19T19:06:14.199 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d42935700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:14.199 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d42935700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c10f270 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d3bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d3bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:60116/0 (socket says 172.21.15.55:60116) 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d3bfff700 1 -- 172.21.15.55:0/809063884 learned_addr learned my addr 172.21.15.55:0/809063884 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.197+0000 7f6d3bfff700 1 -- 172.21.15.55:0/809063884 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24005040 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d3bfff700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b690 tx=0x7f6d24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d24009650 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c10f470 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6d240097b0 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d2401c5b0 con 0x7f6d3c071b60 2023-12-19T19:06:14.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.198+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c110030 con 0x7f6d3c071b60 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.199+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f6d2401c870 con 0x7f6d3c071b60 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.199+0000 7f6d38ff9700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 0x7f6d28038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.199+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6d240498e0 con 0x7f6d3c071b60 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.199+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c05f270 con 0x7f6d3c071b60 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.199+0000 7f6d3b7fe700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 0x7f6d28038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:14.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.200+0000 7f6d3b7fe700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 0x7f6d28038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009990 tx=0x7f6d2c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:14.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.204+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6d24011db0 con 0x7f6d3c071b60 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T19:06:14.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:13 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:06:14.474 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.472+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f6d3c04bb80 con 0x7f6d3c071b60 2023-12-19T19:06:14.477 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.476+0000 7f6d38ff9700 1 -- 172.21.15.55:0/809063884 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f6d240340c0 con 0x7f6d3c071b60 2023-12-19T19:06:14.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 msgr2=0x7f6d28038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:14.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 0x7f6d28038b80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009990 tx=0x7f6d2c006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:14.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 msgr2=0x7f6d3c10ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:14.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b690 tx=0x7f6d24009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:14.480 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 shutdown_connections 2023-12-19T19:06:14.481 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f6d280366d0 0x7f6d28038b80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:14.481 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 --2- 172.21.15.55:0/809063884 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6d3c071b60 0x7f6d3c10ed30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:14.481 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 >> 172.21.15.55:0/809063884 conn(0x7f6d3c0f7f20 msgr2=0x7f6d3c0f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:14.481 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 shutdown_connections 2023-12-19T19:06:14.481 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:14.479+0000 7f6d42935700 1 -- 172.21.15.55:0/809063884 wait complete. 2023-12-19T19:06:15.195 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:14 smithi055 ceph-mon[80978]: Deploying daemon crash.smithi055 on smithi055 2023-12-19T19:06:15.195 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:14 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/809063884' entity='client.admin' 2023-12-19T19:06:15.625 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-19T19:06:15.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-19T19:06:16.557 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:06:18.559 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.556+0000 7f31da846700 1 -- 172.21.15.55:0/1214062406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 msgr2=0x7f31d40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.556+0000 7f31da846700 1 --2- 172.21.15.55:0/1214062406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d40fcc90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f31c40048c0 tx=0x7f31c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.556+0000 7f31da846700 1 -- 172.21.15.55:0/1214062406 shutdown_connections 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 --2- 172.21.15.55:0/1214062406 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d40fcc90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 -- 172.21.15.55:0/1214062406 >> 172.21.15.55:0/1214062406 conn(0x7f31d40f7f40 msgr2=0x7f31d40fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 -- 172.21.15.55:0/1214062406 shutdown_connections 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 -- 172.21.15.55:0/1214062406 wait complete. 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 Processor -- start 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 -- start start 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.557+0000 7f31da846700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31da846700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d410f290 con 0x7f31d40fc880 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:18.560 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42772/0 (socket says 172.21.15.55:42772) 2023-12-19T19:06:18.561 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d3fff700 1 -- 172.21.15.55:0/4020159690 learned_addr learned my addr 172.21.15.55:0/4020159690 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d3fff700 1 -- 172.21.15.55:0/4020159690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c4005040 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d3fff700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f31c400b690 tx=0x7f31c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c4009650 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f31c40097b0 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d410f490 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31c401c5d0 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.558+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d4110050 con 0x7f31d40fc880 2023-12-19T19:06:18.562 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.559+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f31c401c840 con 0x7f31d40fc880 2023-12-19T19:06:18.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.560+0000 7f31d0ff9700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 0x7f31bc038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:18.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.560+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d405f270 con 0x7f31d40fc880 2023-12-19T19:06:18.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.560+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f31c4049c60 con 0x7f31d40fc880 2023-12-19T19:06:18.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.560+0000 7f31d37fe700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 0x7f31bc038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:18.563 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.560+0000 7f31d37fe700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 0x7f31bc038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009990 tx=0x7f31c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:18.570 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.568+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f31c4011d40 con 0x7f31d40fc880 2023-12-19T19:06:18.781 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.778+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f31d4110c30 con 0x7f31bc0366d0 2023-12-19T19:06:18.784 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.782+0000 7f31d0ff9700 1 -- 172.21.15.55:0/4020159690 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31d4110c30 con 0x7f31bc0366d0 2023-12-19T19:06:18.786 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 msgr2=0x7f31bc038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 0x7f31bc038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f31c8009990 tx=0x7f31c8006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 msgr2=0x7f31d410ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f31c400b690 tx=0x7f31c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 shutdown_connections 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f31bc0366d0 0x7f31bc038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 --2- 172.21.15.55:0/4020159690 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f31d40fc880 0x7f31d410ed50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 >> 172.21.15.55:0/4020159690 conn(0x7f31d40f7f40 msgr2=0x7f31d40fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 shutdown_connections 2023-12-19T19:06:18.787 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:18.784+0000 7f31da846700 1 -- 172.21.15.55:0/4020159690 wait complete. 2023-12-19T19:06:19.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:19.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:19.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:19.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:19.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: Deploying daemon node-exporter.smithi055 on smithi055 2023-12-19T19:06:19.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:06:19.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:19 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:19.356 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2023-12-19T19:06:19.356 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:06:19.357 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2023-12-19T19:06:19.375 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:06:19.375 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:06:19.433 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2023-12-19T19:06:19.433 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch host add smithi187 2023-12-19T19:06:19.698 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 -- 172.21.15.55:0/1191741801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 msgr2=0x7f3e340fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 --2- 172.21.15.55:0/1191741801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e340fcc90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e300048c0 tx=0x7f3e30009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 -- 172.21.15.55:0/1191741801 shutdown_connections 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 --2- 172.21.15.55:0/1191741801 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e340fcc90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 -- 172.21.15.55:0/1191741801 >> 172.21.15.55:0/1191741801 conn(0x7f3e340f7f40 msgr2=0x7f3e340fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.972+0000 7f3e3be90700 1 -- 172.21.15.55:0/1191741801 shutdown_connections 2023-12-19T19:06:22.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e3be90700 1 -- 172.21.15.55:0/1191741801 wait complete. 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e3be90700 1 Processor -- start 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e3be90700 1 -- start start 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e3be90700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e3be90700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3410d010 con 0x7f3e340fc880 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e39c2c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e39c2c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42788/0 (socket says 172.21.15.55:42788) 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e39c2c700 1 -- 172.21.15.55:0/2017388372 learned_addr learned my addr 172.21.15.55:0/2017388372 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.973+0000 7f3e39c2c700 1 -- 172.21.15.55:0/2017388372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30005040 con 0x7f3e340fc880 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e39c2c700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000b690 tx=0x7f3e30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e30009650 con 0x7f3e340fc880 2023-12-19T19:06:22.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e300097b0 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3410d210 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e3001c5d0 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.974+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3410deb0 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f3e300139d0 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3405f270 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e267fc700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 0x7f3e20038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3e30049920 con 0x7f3e340fc880 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e3942b700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 0x7f3e20038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.975+0000 7f3e3942b700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 0x7f3e20038bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009990 tx=0x7f3e28006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:22.023 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:21.980+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3e30025400 con 0x7f3e340fc880 2023-12-19T19:06:22.185 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:22.182+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e3410ea70 con 0x7f3e20036720 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: Deploying daemon alertmanager.smithi055 on smithi055 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: Deploying cephadm binary to smithi187 2023-12-19T19:06:24.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:23 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:24.171 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.169+0000 7f3e267fc700 1 -- 172.21.15.55:0/2017388372 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f3e3410ea70 con 0x7f3e20036720 2023-12-19T19:06:24.171 INFO:teuthology.orchestra.run.smithi055.stdout:Added host 'smithi187' with addr '172.21.15.187' 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.171+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 msgr2=0x7f3e20038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.171+0000 7f3e3be90700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 0x7f3e20038bd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009990 tx=0x7f3e28006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.171+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 msgr2=0x7f3e3410cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.171+0000 7f3e3be90700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000b690 tx=0x7f3e30009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 shutdown_connections 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f3e20036720 0x7f3e20038bd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 --2- 172.21.15.55:0/2017388372 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3e340fc880 0x7f3e3410cad0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 >> 172.21.15.55:0/2017388372 conn(0x7f3e340f7f40 msgr2=0x7f3e340f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 shutdown_connections 2023-12-19T19:06:24.174 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:24.172+0000 7f3e3be90700 1 -- 172.21.15.55:0/2017388372 wait complete. 2023-12-19T19:06:24.997 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-19T19:06:25.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:25 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:25.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:25 smithi055 ceph-mon[80978]: Added host smithi187 2023-12-19T19:06:25.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:25 smithi055 ceph-mon[80978]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:25.418 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:06:28.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:27 smithi055 ceph-mon[80978]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 -- 172.21.15.55:0/2863584849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 msgr2=0x7f0e000ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/2863584849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e000ff460 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0048c0 tx=0x7f0dfc009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 -- 172.21.15.55:0/2863584849 shutdown_connections 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/2863584849 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e000ff460 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 -- 172.21.15.55:0/2863584849 >> 172.21.15.55:0/2863584849 conn(0x7f0e000f6e50 msgr2=0x7f0e000f9280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 -- 172.21.15.55:0/2863584849 shutdown_connections 2023-12-19T19:06:28.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.672+0000 7f0e07b1f700 1 -- 172.21.15.55:0/2863584849 wait complete. 2023-12-19T19:06:28.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.673+0000 7f0e07b1f700 1 Processor -- start 2023-12-19T19:06:28.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.673+0000 7f0e07b1f700 1 -- start start 2023-12-19T19:06:28.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.673+0000 7f0e07b1f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:28.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.673+0000 7f0e07b1f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0010cfc0 con 0x7f0e00066590 2023-12-19T19:06:28.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e058bb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:28.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e058bb700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48334/0 (socket says 172.21.15.55:48334) 2023-12-19T19:06:28.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e058bb700 1 -- 172.21.15.55:0/798381638 learned_addr learned my addr 172.21.15.55:0/798381638 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e058bb700 1 -- 172.21.15.55:0/798381638 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc005040 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e058bb700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009040 tx=0x7f0dfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dfc009620 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0010d1c0 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dfc009780 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dfc01c5b0 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.674+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0010de60 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.675+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f0dfc01c870 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.675+0000 7f0df27fc700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 0x7f0dec038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.675+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e0005f270 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.675+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0dfc0498c0 con 0x7f0e00066590 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.675+0000 7f0e050ba700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 0x7f0dec038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:28.680 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.676+0000 7f0e050ba700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 0x7f0dec038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009990 tx=0x7f0df4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:28.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.680+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0dfc023300 con 0x7f0e00066590 2023-12-19T19:06:28.886 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.884+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e0010e9f0 con 0x7f0dec0366d0 2023-12-19T19:06:28.887 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.885+0000 7f0df27fc700 1 -- 172.21.15.55:0/798381638 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f0e0010e9f0 con 0x7f0dec0366d0 2023-12-19T19:06:28.888 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:06:28.888 INFO:teuthology.orchestra.run.smithi055.stdout:[{"addr": "172.21.15.55", "hostname": "smithi055", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2023-12-19T19:06:28.889 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 msgr2=0x7f0dec038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 0x7f0dec038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009990 tx=0x7f0df4006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 msgr2=0x7f0e0010ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc009040 tx=0x7f0dfc009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 shutdown_connections 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0dec0366d0 0x7f0dec038b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 --2- 172.21.15.55:0/798381638 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0e00066590 0x7f0e0010ca80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 >> 172.21.15.55:0/798381638 conn(0x7f0e000f6e50 msgr2=0x7f0e000f7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 shutdown_connections 2023-12-19T19:06:28.890 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:28.887+0000 7f0e07b1f700 1 -- 172.21.15.55:0/798381638 wait complete. 2023-12-19T19:06:30.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:29 smithi055 ceph-mon[80978]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:30.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:29 smithi055 ceph-mon[80978]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:06:30.953 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-19T19:06:30.953 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-19T19:06:31.772 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T19:06:32.104 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:31 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:33.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:32 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T19:06:33.036 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:32 smithi055 ceph-mon[80978]: Deploying daemon grafana.smithi055 on smithi055 2023-12-19T19:06:33.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 -- 172.21.15.55:0/1911035121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 msgr2=0x7fe004101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:33.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 --2- 172.21.15.55:0/1911035121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe004101850 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fdff80048c0 tx=0x7fdff8009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:33.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 -- 172.21.15.55:0/1911035121 shutdown_connections 2023-12-19T19:06:33.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 --2- 172.21.15.55:0/1911035121 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe004101850 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:33.404 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 -- 172.21.15.55:0/1911035121 >> 172.21.15.55:0/1911035121 conn(0x7fe0040f6e40 msgr2=0x7fe0040f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 -- 172.21.15.55:0/1911035121 shutdown_connections 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.239+0000 7fe009533700 1 -- 172.21.15.55:0/1911035121 wait complete. 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe009533700 1 Processor -- start 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe009533700 1 -- start start 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe009533700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe009533700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00410f230 con 0x7fe004065a80 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe002ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe002ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48340/0 (socket says 172.21.15.55:48340) 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe002ffd700 1 -- 172.21.15.55:0/1756681708 learned_addr learned my addr 172.21.15.55:0/1756681708 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe002ffd700 1 -- 172.21.15.55:0/1756681708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff8005040 con 0x7fe004065a80 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fe002ffd700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdff800b690 tx=0x7fdff8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.240+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff8009650 con 0x7fe004065a80 2023-12-19T19:06:33.405 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.241+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdff80097b0 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.241+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00410f430 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.241+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdff801c5b0 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.241+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00410ff90 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fdff801c820 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00404bb80 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fdfeb7fe700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 0x7fdfec038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdff8049930 con 0x7fe004065a80 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fe0027fc700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 0x7fdfec038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.242+0000 7fe0027fc700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 0x7fdfec038bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009990 tx=0x7fdff4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:33.406 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.247+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdff8013600 con 0x7fe004065a80 2023-12-19T19:06:33.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.448+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe004110e10 con 0x7fe004065a80 2023-12-19T19:06:33.856 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.853+0000 7fdfeb7fe700 1 -- 172.21.15.55:0/1756681708 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdff801b390 con 0x7fe004065a80 2023-12-19T19:06:33.858 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 msgr2=0x7fdfec038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:33.858 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 0x7fdfec038bd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdff4009990 tx=0x7fdff4006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:33.858 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 msgr2=0x7fe00410ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:33.858 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdff800b690 tx=0x7fdff8009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 shutdown_connections 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fdfec036720 0x7fdfec038bd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 --2- 172.21.15.55:0/1756681708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe004065a80 0x7fe00410ecf0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 >> 172.21.15.55:0/1756681708 conn(0x7fe0040f6e40 msgr2=0x7fe0040f7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 shutdown_connections 2023-12-19T19:06:33.859 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:06:33.856+0000 7fe009533700 1 -- 172.21.15.55:0/1756681708 wait complete. 2023-12-19T19:06:33.860 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2023-12-19T19:06:34.234 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:33 smithi055 ceph-mon[80978]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:34.234 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:33 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:34.234 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:33 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1756681708' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-19T19:06:34.559 INFO:tasks.cephadm:Adding mon.smithi055 on smithi055 2023-12-19T19:06:34.559 INFO:tasks.cephadm:Adding mon.smithi187 on smithi187 2023-12-19T19:06:34.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187' 2023-12-19T19:06:34.806 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:34.852 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:34 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1756681708' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-19T19:06:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:34 smithi055 ceph-mon[80978]: osdmap e4: 0 total, 0 up, 0 in 2023-12-19T19:06:36.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:35 smithi055 ceph-mon[80978]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- 172.21.15.187:0/2174322350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 msgr2=0x7f72280fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 --2- 172.21.15.187:0/2174322350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f72280fcc70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f721c0048c0 tx=0x7f721c009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- 172.21.15.187:0/2174322350 shutdown_connections 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 --2- 172.21.15.187:0/2174322350 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f72280fcc70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- 172.21.15.187:0/2174322350 >> 172.21.15.187:0/2174322350 conn(0x7f72280f7fe0 msgr2=0x7f72280fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- 172.21.15.187:0/2174322350 shutdown_connections 2023-12-19T19:06:36.389 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- 172.21.15.187:0/2174322350 wait complete. 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 Processor -- start 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.386+0000 7f722f129700 1 -- start start 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722f129700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722f129700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722810cfc0 con 0x7f72280fc860 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722cec5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722cec5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:42316/0 (socket says 172.21.15.187:42316) 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722cec5700 1 -- 172.21.15.187:0/3255484839 learned_addr learned my addr 172.21.15.187:0/3255484839 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:36.390 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722cec5700 1 -- 172.21.15.187:0/3255484839 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c005040 con 0x7f72280fc860 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722cec5700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f721c00b690 tx=0x7f721c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c009650 con 0x7f72280fc860 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f721c0097b0 con 0x7f72280fc860 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f721c01c5b0 con 0x7f72280fc860 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.387+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722810d1c0 con 0x7f72280fc860 2023-12-19T19:06:36.391 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.388+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722810de60 con 0x7f72280fc860 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f721c01c820 con 0x7f72280fc860 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722805f270 con 0x7f72280fc860 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f72257fa700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 0x7f7218038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f721c049810 con 0x7f72280fc860 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f7227fff700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 0x7f7218038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:36.392 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.389+0000 7f7227fff700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 0x7f7218038bd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7214009990 tx=0x7f7214006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:36.396 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.394+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f721c011550 con 0x7f72280fc860 2023-12-19T19:06:36.588 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.585+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 --> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187", "target": ["mon-mgr", ""]}) v1 -- 0x7f722810ead0 con 0x7f7218036720 2023-12-19T19:06:36.594 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.592+0000 7f72257fa700 1 -- 172.21.15.187:0/3255484839 <== mgr.14162 v2:172.21.15.55:6800/797433438 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f722810ead0 con 0x7f7218036720 2023-12-19T19:06:36.595 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2023-12-19T19:06:36.596 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 msgr2=0x7f7218038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:36.596 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 0x7f7218038bd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7214009990 tx=0x7f7214006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:36.596 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 msgr2=0x7f722810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f721c00b690 tx=0x7f721c009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 shutdown_connections 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f7218036720 0x7f7218038bd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 --2- 172.21.15.187:0/3255484839 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f72280fc860 0x7f722810ca80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 >> 172.21.15.187:0/3255484839 conn(0x7f72280f7fe0 msgr2=0x7f72280f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 shutdown_connections 2023-12-19T19:06:36.597 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:36.594+0000 7f722f129700 1 -- 172.21.15.187:0/3255484839 wait complete. 2023-12-19T19:06:37.129 DEBUG:teuthology.orchestra.run.smithi187:mon.smithi187> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi187.service 2023-12-19T19:06:37.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:37.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:37.449 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:38.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:37 smithi055 ceph-mon[80978]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:06:38.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:37 smithi055 ceph-mon[80978]: Saving service mon spec with placement smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187;count:2 2023-12-19T19:06:38.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:37 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:38.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:37 smithi055 ceph-mon[80978]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:38.904 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- 172.21.15.187:0/409402694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 msgr2=0x7f0b800fc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 --2- 172.21.15.187:0/409402694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b800fc150 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0b740048c0 tx=0x7f0b74009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- 172.21.15.187:0/409402694 shutdown_connections 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 --2- 172.21.15.187:0/409402694 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b800fc150 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- 172.21.15.187:0/409402694 >> 172.21.15.187:0/409402694 conn(0x7f0b800f65d0 msgr2=0x7f0b800f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- 172.21.15.187:0/409402694 shutdown_connections 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- 172.21.15.187:0/409402694 wait complete. 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 Processor -- start 2023-12-19T19:06:38.905 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.901+0000 7f0b86b57700 1 -- start start 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b86b57700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b86b57700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b800fb670 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b848f3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b848f3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:42324/0 (socket says 172.21.15.187:42324) 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b848f3700 1 -- 172.21.15.187:0/3617469307 learned_addr learned my addr 172.21.15.187:0/3617469307 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b848f3700 1 -- 172.21.15.187:0/3617469307 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b74005040 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b848f3700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009330 tx=0x7f0b74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b740096b0 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b74009810 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b800fb870 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.902+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b7401c5b0 con 0x7f0b800fbd40 2023-12-19T19:06:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.903+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8010ff80 con 0x7f0b800fbd40 2023-12-19T19:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.903+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f0b74013420 con 0x7f0b800fbd40 2023-12-19T19:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.903+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b80105400 con 0x7f0b800fbd40 2023-12-19T19:06:38.907 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.903+0000 7f0b7d7fa700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 0x7f0b70038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.904+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0b74049ad0 con 0x7f0b800fbd40 2023-12-19T19:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.904+0000 7f0b7ffff700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 0x7f0b70038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:38.908 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.904+0000 7f0b7ffff700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 0x7f0b70038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009990 tx=0x7f0b6c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:38.911 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:38.909+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0b740119c0 con 0x7f0b800fbd40 2023-12-19T19:06:39.171 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.169+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b801013f0 con 0x7f0b800fbd40 2023-12-19T19:06:39.172 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.170+0000 7f0b7d7fa700 1 -- 172.21.15.187:0/3617469307 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0b74011370 con 0x7f0b800fbd40 2023-12-19T19:06:39.172 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:39.172 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 msgr2=0x7f0b70038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 0x7f0b70038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c009990 tx=0x7f0b6c006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 msgr2=0x7f0b8010f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009330 tx=0x7f0b74009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 shutdown_connections 2023-12-19T19:06:39.174 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0b700366d0 0x7f0b70038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 --2- 172.21.15.187:0/3617469307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0b800fbd40 0x7f0b8010f800 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 >> 172.21.15.187:0/3617469307 conn(0x7f0b800f65d0 msgr2=0x7f0b800f7400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 shutdown_connections 2023-12-19T19:06:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:39.172+0000 7f0b86b57700 1 -- 172.21.15.187:0/3617469307 wait complete. 2023-12-19T19:06:39.175 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:40.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:39 smithi055 ceph-mon[80978]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:40.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:39 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/3617469307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:40.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:40.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:41.021 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:41.067 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:42.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:41 smithi055 ceph-mon[80978]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 -- 172.21.15.187:0/2251742220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 msgr2=0x7fa20c0faf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 --2- 172.21.15.187:0/2251742220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c0faf40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa2080048c0 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 -- 172.21.15.187:0/2251742220 shutdown_connections 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 --2- 172.21.15.187:0/2251742220 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c0faf40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 -- 172.21.15.187:0/2251742220 >> 172.21.15.187:0/2251742220 conn(0x7fa20c0f65d0 msgr2=0x7fa20c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 -- 172.21.15.187:0/2251742220 shutdown_connections 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 -- 172.21.15.187:0/2251742220 wait complete. 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.563+0000 7fa2143f5700 1 Processor -- start 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa2143f5700 1 -- start start 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa2143f5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa2143f5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20c10cfb0 con 0x7fa20c0fab30 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa212191700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:42.568 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa212191700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:53728/0 (socket says 172.21.15.187:53728) 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa212191700 1 -- 172.21.15.187:0/187273837 learned_addr learned my addr 172.21.15.187:0/187273837 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa212191700 1 -- 172.21.15.187:0/187273837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa208005040 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.564+0000 7fa212191700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa208009040 tx=0x7fa208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa208009650 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20c10d1b0 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa2080097b0 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20c10de50 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa208025560 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fa208013420 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.565+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa20c05f270 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.566+0000 7fa202ffd700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 0x7fa1f8038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.566+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa208049990 con 0x7fa20c0fab30 2023-12-19T19:06:42.569 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.566+0000 7fa211990700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 0x7fa1f8038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:42.570 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.566+0000 7fa211990700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 0x7fa1f8038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009990 tx=0x7fa1fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:42.573 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.571+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa2080139d0 con 0x7fa20c0fab30 2023-12-19T19:06:42.833 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.831+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa20c10e9a0 con 0x7fa20c0fab30 2023-12-19T19:06:42.833 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.831+0000 7fa202ffd700 1 -- 172.21.15.187:0/187273837 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa208011370 con 0x7fa20c0fab30 2023-12-19T19:06:42.834 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:42.834 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 msgr2=0x7fa1f8038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 0x7fa1f8038b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc009990 tx=0x7fa1fc006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 msgr2=0x7fa20c10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa208009040 tx=0x7fa208009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 shutdown_connections 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fa1f80366d0 0x7fa1f8038b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 --2- 172.21.15.187:0/187273837 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa20c0fab30 0x7fa20c10ca70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:42.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 >> 172.21.15.187:0/187273837 conn(0x7fa20c0f65d0 msgr2=0x7fa20c0f7370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:42.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 shutdown_connections 2023-12-19T19:06:42.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:42.834+0000 7fa2143f5700 1 -- 172.21.15.187:0/187273837 wait complete. 2023-12-19T19:06:42.837 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:43.719 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:43 smithi055 ceph-mon[80978]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:43.719 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:43 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/187273837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:44.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:44.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:44.664 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:44.711 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:46.080 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- 172.21.15.187:0/1811885653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 msgr2=0x7f487c0fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 --2- 172.21.15.187:0/1811885653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c0fcc90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48700048c0 tx=0x7f4870009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- 172.21.15.187:0/1811885653 shutdown_connections 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 --2- 172.21.15.187:0/1811885653 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c0fcc90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- 172.21.15.187:0/1811885653 >> 172.21.15.187:0/1811885653 conn(0x7f487c0f7f40 msgr2=0x7f487c0fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- 172.21.15.187:0/1811885653 shutdown_connections 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- 172.21.15.187:0/1811885653 wait complete. 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 Processor -- start 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- start start 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.077+0000 7f4882f09700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c10d000 con 0x7f487c0fc880 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4880ca5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4880ca5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:53748/0 (socket says 172.21.15.187:53748) 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4880ca5700 1 -- 172.21.15.187:0/749057785 learned_addr learned my addr 172.21.15.187:0/749057785 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:46.081 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4880ca5700 1 -- 172.21.15.187:0/749057785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870005040 con 0x7f487c0fc880 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4880ca5700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4870009330 tx=0x7f4870009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48700096b0 con 0x7f487c0fc880 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c10d200 con 0x7f487c0fc880 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4870009810 con 0x7f487c0fc880 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f487001c5b0 con 0x7f487c0fc880 2023-12-19T19:06:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.078+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c10dea0 con 0x7f487c0fc880 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.079+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4870013420 con 0x7f487c0fc880 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.079+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c05f270 con 0x7f487c0fc880 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.080+0000 7f48797fa700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 0x7f486c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.080+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4870049a00 con 0x7f487c0fc880 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.080+0000 7f487bfff700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 0x7f486c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:46.083 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.080+0000 7f487bfff700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 0x7f486c038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4868009990 tx=0x7f4868006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:46.086 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.085+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f48700119c0 con 0x7f487c0fc880 2023-12-19T19:06:46.139 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:45 smithi055 ceph-mon[80978]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:46.346 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.344+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f487c10e9f0 con 0x7f487c0fc880 2023-12-19T19:06:46.347 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.344+0000 7f48797fa700 1 -- 172.21.15.187:0/749057785 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4870011370 con 0x7f487c0fc880 2023-12-19T19:06:46.347 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:46.347 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 msgr2=0x7f486c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 0x7f486c038b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4868009990 tx=0x7f4868006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 msgr2=0x7f487c10cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4870009330 tx=0x7f4870009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 shutdown_connections 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f486c0366d0 0x7f486c038b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 --2- 172.21.15.187:0/749057785 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f487c0fc880 0x7f487c10cac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 >> 172.21.15.187:0/749057785 conn(0x7f487c0f7f40 msgr2=0x7f487c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:46.349 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 shutdown_connections 2023-12-19T19:06:46.350 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:46.347+0000 7f4882f09700 1 -- 172.21.15.187:0/749057785 wait complete. 2023-12-19T19:06:46.350 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/749057785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:46.956 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:46 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:47.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:47.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:48.199 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:48.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:47 smithi055 ceph-mon[80978]: Deploying daemon prometheus.smithi055 on smithi055 2023-12-19T19:06:48.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:47 smithi055 ceph-mon[80978]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:49.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:48 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 -- 172.21.15.187:0/4084212392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 msgr2=0x7fc4c80fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/4084212392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c80fcc80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4b80048c0 tx=0x7fc4b8009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 -- 172.21.15.187:0/4084212392 shutdown_connections 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/4084212392 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c80fcc80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 -- 172.21.15.187:0/4084212392 >> 172.21.15.187:0/4084212392 conn(0x7fc4c80f7f50 msgr2=0x7fc4c80fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 -- 172.21.15.187:0/4084212392 shutdown_connections 2023-12-19T19:06:49.669 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 -- 172.21.15.187:0/4084212392 wait complete. 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.666+0000 7fc4cdae8700 1 Processor -- start 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4cdae8700 1 -- start start 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4cdae8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4cdae8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c810d020 con 0x7fc4c80fc870 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:55012/0 (socket says 172.21.15.187:55012) 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c77fe700 1 -- 172.21.15.187:0/2797835074 learned_addr learned my addr 172.21.15.187:0/2797835074 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:49.670 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c77fe700 1 -- 172.21.15.187:0/2797835074 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4b8005040 con 0x7fc4c80fc870 2023-12-19T19:06:49.671 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c77fe700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009330 tx=0x7fc4b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:49.671 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b80096b0 con 0x7fc4c80fc870 2023-12-19T19:06:49.671 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4b8009810 con 0x7fc4c80fc870 2023-12-19T19:06:49.671 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c810d220 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4b801c5b0 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.667+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c810dec0 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.668+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fc4b8013420 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.668+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c805f270 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.668+0000 7fc4c4ff9700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 0x7fc4b4038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.669+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc4b8049ac0 con 0x7fc4c80fc870 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.669+0000 7fc4beffd700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 0x7fc4b4038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:49.672 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.669+0000 7fc4beffd700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 0x7fc4b4038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009990 tx=0x7fc4b0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:49.678 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.676+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc4b8011db0 con 0x7fc4c80fc870 2023-12-19T19:06:49.940 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.938+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4c810ea10 con 0x7fc4c80fc870 2023-12-19T19:06:49.941 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.939+0000 7fc4c4ff9700 1 -- 172.21.15.187:0/2797835074 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc4b8011370 con 0x7fc4c80fc870 2023-12-19T19:06:49.941 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:49.941 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.940+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 msgr2=0x7fc4b4038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.940+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 0x7fc4b4038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0009990 tx=0x7fc4b0006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.940+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 msgr2=0x7fc4c810cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.940+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009330 tx=0x7fc4b8009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 shutdown_connections 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7fc4b40366d0 0x7fc4b4038b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 --2- 172.21.15.187:0/2797835074 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4c80fc870 0x7fc4c810cae0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 >> 172.21.15.187:0/2797835074 conn(0x7fc4c80f7f50 msgr2=0x7fc4c80f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 shutdown_connections 2023-12-19T19:06:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:49.941+0000 7fc4cdae8700 1 -- 172.21.15.187:0/2797835074 wait complete. 2023-12-19T19:06:49.944 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:50.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:49 smithi055 ceph-mon[80978]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:51.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:50 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2797835074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:51.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:51.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:51.819 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:51.866 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:52.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:51 smithi055 ceph-mon[80978]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- 172.21.15.187:0/231953446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 msgr2=0x7f91c00fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/231953446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c00fbb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f91b40048c0 tx=0x7f91b4009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- 172.21.15.187:0/231953446 shutdown_connections 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/231953446 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c00fbb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- 172.21.15.187:0/231953446 >> 172.21.15.187:0/231953446 conn(0x7f91c00f6e50 msgr2=0x7f91c00f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:53.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- 172.21.15.187:0/231953446 shutdown_connections 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- 172.21.15.187:0/231953446 wait complete. 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 Processor -- start 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.265+0000 7f91c6dbd700 1 -- start start 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c6dbd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c6dbd700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c010bf00 con 0x7f91c00fb770 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c4b59700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c4b59700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:55022/0 (socket says 172.21.15.187:55022) 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c4b59700 1 -- 172.21.15.187:0/1547484737 learned_addr learned my addr 172.21.15.187:0/1547484737 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:53.269 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c4b59700 1 -- 172.21.15.187:0/1547484737 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b4005040 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.266+0000 7f91c4b59700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f91b400b690 tx=0x7f91b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b4009650 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c010c100 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91b40097b0 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c010cda0 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91b401c5d0 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.267+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f91b401c840 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.268+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c005f270 con 0x7f91c00fb770 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.268+0000 7f91bd7fa700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 0x7f91b0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:53.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.268+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f91b4049a50 con 0x7f91c00fb770 2023-12-19T19:06:53.272 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.268+0000 7f91bffff700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 0x7f91b0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:53.272 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.268+0000 7f91bffff700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 0x7f91b0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009990 tx=0x7f91ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:53.274 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.273+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f91b4011d40 con 0x7f91c00fb770 2023-12-19T19:06:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.532+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91c010d910 con 0x7f91c00fb770 2023-12-19T19:06:53.535 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.533+0000 7f91bd7fa700 1 -- 172.21.15.187:0/1547484737 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f91b4011b60 con 0x7f91c00fb770 2023-12-19T19:06:53.535 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:53.535 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:53.537 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 msgr2=0x7f91b0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:53.537 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 0x7f91b0038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009990 tx=0x7f91ac006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 msgr2=0x7f91c010b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f91b400b690 tx=0x7f91b4009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 shutdown_connections 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f91b00366d0 0x7f91b0038b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 --2- 172.21.15.187:0/1547484737 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f91c00fb770 0x7f91c010b9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 >> 172.21.15.187:0/1547484737 conn(0x7f91c00f6e50 msgr2=0x7f91c00f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 shutdown_connections 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:53.535+0000 7f91c6dbd700 1 -- 172.21.15.187:0/1547484737 wait complete. 2023-12-19T19:06:53.538 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:53.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:53 smithi055 ceph-mon[80978]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:53.814 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:53 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/1547484737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:55.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:55.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:55.372 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:55.419 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:56.138 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:55 smithi055 ceph-mon[80978]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:56.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 -- 172.21.15.187:0/2933251519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 msgr2=0x7f05140ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:56.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 --2- 172.21.15.187:0/2933251519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f05140ff440 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f050c0048c0 tx=0x7f050c009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:56.837 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 -- 172.21.15.187:0/2933251519 shutdown_connections 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 --2- 172.21.15.187:0/2933251519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f05140ff440 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 -- 172.21.15.187:0/2933251519 >> 172.21.15.187:0/2933251519 conn(0x7f05140f6e50 msgr2=0x7f05140f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 -- 172.21.15.187:0/2933251519 shutdown_connections 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.833+0000 7f051badf700 1 -- 172.21.15.187:0/2933251519 wait complete. 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051badf700 1 Processor -- start 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051badf700 1 -- start start 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051badf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051badf700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051410cfe0 con 0x7f0514066510 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051987b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051987b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:55050/0 (socket says 172.21.15.187:55050) 2023-12-19T19:06:56.838 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.834+0000 7f051987b700 1 -- 172.21.15.187:0/2189299774 learned_addr learned my addr 172.21.15.187:0/2189299774 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f051987b700 1 -- 172.21.15.187:0/2189299774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c005040 con 0x7f0514066510 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f051987b700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f050c00b690 tx=0x7f050c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f050c009650 con 0x7f0514066510 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f050c0097b0 con 0x7f0514066510 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051410d1e0 con 0x7f0514066510 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f050c01c5b0 con 0x7f0514066510 2023-12-19T19:06:56.839 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.835+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051410de80 con 0x7f0514066510 2023-12-19T19:06:56.841 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.836+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f050c013420 con 0x7f0514066510 2023-12-19T19:06:56.841 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.836+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051405f270 con 0x7f0514066510 2023-12-19T19:06:56.841 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.836+0000 7f05127fc700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 0x7f04fc038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:06:56.841 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.836+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f050c049910 con 0x7f0514066510 2023-12-19T19:06:56.841 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.836+0000 7f051907a700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 0x7f04fc038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:06:56.842 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.837+0000 7f051907a700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 0x7f04fc038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0508009990 tx=0x7f0508006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:06:56.843 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:56.841+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f050c013a60 con 0x7f0514066510 2023-12-19T19:06:57.098 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.096+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f051410e9d0 con 0x7f0514066510 2023-12-19T19:06:57.099 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.097+0000 7f05127fc700 1 -- 172.21.15.187:0/2189299774 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f050c011370 con 0x7f0514066510 2023-12-19T19:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 msgr2=0x7f04fc038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 0x7f04fc038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0508009990 tx=0x7f0508006e40 comp rx=0 tx=0).stop 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 msgr2=0x7f051410caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f050c00b690 tx=0x7f050c009e30 comp rx=0 tx=0).stop 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 shutdown_connections 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f04fc0366d0 0x7f04fc038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 --2- 172.21.15.187:0/2189299774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0514066510 0x7f051410caa0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 >> 172.21.15.187:0/2189299774 conn(0x7f05140f6e50 msgr2=0x7f05140f7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 shutdown_connections 2023-12-19T19:06:57.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:06:57.099+0000 7f051badf700 1 -- 172.21.15.187:0/2189299774 wait complete. 2023-12-19T19:06:57.103 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2189299774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:06:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:57 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:06:58.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:06:58.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:06:58.922 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:58.969 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:06:59.244 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:58 smithi055 ceph-mon[80978]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-19T19:06:59.244 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:06:58 smithi055 ceph-mon[80978]: mgrmap e14: smithi055.dwclvp(active, since 55s) 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.460+0000 7f1b1e439700 1 -- 172.21.15.187:0/1490734071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 msgr2=0x7f1b180fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.460+0000 7f1b1e439700 1 --2- 172.21.15.187:0/1490734071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b180fcc90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1b080048c0 tx=0x7f1b08009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.460+0000 7f1b1e439700 1 -- 172.21.15.187:0/1490734071 shutdown_connections 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.460+0000 7f1b1e439700 1 --2- 172.21.15.187:0/1490734071 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b180fcc90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.460+0000 7f1b1e439700 1 -- 172.21.15.187:0/1490734071 >> 172.21.15.187:0/1490734071 conn(0x7f1b180f7f40 msgr2=0x7f1b180fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.461+0000 7f1b1e439700 1 -- 172.21.15.187:0/1490734071 shutdown_connections 2023-12-19T19:07:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.461+0000 7f1b1e439700 1 -- 172.21.15.187:0/1490734071 wait complete. 2023-12-19T19:07:00.464 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.461+0000 7f1b1e439700 1 Processor -- start 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.461+0000 7f1b1e439700 1 -- start start 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b1e439700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b1e439700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1810d000 con 0x7f1b180fc880 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b17fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b17fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:35104/0 (socket says 172.21.15.187:35104) 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b17fff700 1 -- 172.21.15.187:0/3250985082 learned_addr learned my addr 172.21.15.187:0/3250985082 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b17fff700 1 -- 172.21.15.187:0/3250985082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b08005040 con 0x7f1b180fc880 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b17fff700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009330 tx=0x7f1b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b080096b0 con 0x7f1b180fc880 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1810d200 con 0x7f1b180fc880 2023-12-19T19:07:00.465 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b08009810 con 0x7f1b180fc880 2023-12-19T19:07:00.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b0801c5b0 con 0x7f1b180fc880 2023-12-19T19:07:00.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.462+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1810dea0 con 0x7f1b180fc880 2023-12-19T19:07:00.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f1b0801c870 con 0x7f1b180fc880 2023-12-19T19:07:00.466 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1805f270 con 0x7f1b180fc880 2023-12-19T19:07:00.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b14ff9700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 0x7f1b00038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:00.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1b08049c50 con 0x7f1b180fc880 2023-12-19T19:07:00.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b177fe700 1 -- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 msgr2=0x7f1b00038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/797433438 2023-12-19T19:07:00.467 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.463+0000 7f1b177fe700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 0x7f1b00038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:07:00.470 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.468+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1b08011bd0 con 0x7f1b180fc880 2023-12-19T19:07:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.664+0000 7f1b177fe700 1 -- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 msgr2=0x7f1b00038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/797433438 2023-12-19T19:07:00.666 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.664+0000 7f1b177fe700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 0x7f1b00038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:07:00.731 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.728+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b1810e9f0 con 0x7f1b180fc880 2023-12-19T19:07:00.731 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.729+0000 7f1b14ff9700 1 -- 172.21.15.187:0/3250985082 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1b080117e0 con 0x7f1b180fc880 2023-12-19T19:07:00.731 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:00.731 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:00.733 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 msgr2=0x7f1b00038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 0x7f1b00038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 msgr2=0x7f1b1810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009330 tx=0x7f1b08009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 shutdown_connections 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f1b00036720 0x7f1b00038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 --2- 172.21.15.187:0/3250985082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1b180fc880 0x7f1b1810cac0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 >> 172.21.15.187:0/3250985082 conn(0x7f1b180f7f40 msgr2=0x7f1b180f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 shutdown_connections 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:00.731+0000 7f1b1e439700 1 -- 172.21.15.187:0/3250985082 wait complete. 2023-12-19T19:07:00.734 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:01.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:00 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/3250985082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:02.366 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:02.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:02.614 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:07:02.660 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- 172.21.15.187:0/3072473811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 msgr2=0x7f0bc80faf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 --2- 172.21.15.187:0/3072473811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc80faf40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40048c0 tx=0x7f0bc4009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- 172.21.15.187:0/3072473811 shutdown_connections 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 --2- 172.21.15.187:0/3072473811 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc80faf40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- 172.21.15.187:0/3072473811 >> 172.21.15.187:0/3072473811 conn(0x7f0bc80f65d0 msgr2=0x7f0bc80f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- 172.21.15.187:0/3072473811 shutdown_connections 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- 172.21.15.187:0/3072473811 wait complete. 2023-12-19T19:07:04.088 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 Processor -- start 2023-12-19T19:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.085+0000 7f0bd022a700 1 -- start start 2023-12-19T19:07:04.116 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bd022a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bd022a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810cff0 con 0x7f0bc80fab30 2023-12-19T19:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bcdfc6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bcdfc6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:35114/0 (socket says 172.21.15.187:35114) 2023-12-19T19:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bcdfc6700 1 -- 172.21.15.187:0/2974793105 learned_addr learned my addr 172.21.15.187:0/2974793105 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:04.117 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bcdfc6700 1 -- 172.21.15.187:0/2974793105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4005040 con 0x7f0bc80fab30 2023-12-19T19:07:04.118 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bcdfc6700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009040 tx=0x7f0bc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:04.118 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc4009650 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc810d1f0 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bc40097b0 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bc401c5b0 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.086+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc810de90 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.087+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f0bc4013420 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.087+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc805f270 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.087+0000 7f0bbe7fc700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 0x7f0bb4038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.087+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0bc40499d0 con 0x7f0bc80fab30 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.088+0000 7f0bcd7c5700 1 -- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 msgr2=0x7f0bb4038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/797433438 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.088+0000 7f0bcd7c5700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 0x7f0bb4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:07:04.119 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.092+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0bc40139d0 con 0x7f0bc80fab30 2023-12-19T19:07:04.291 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.289+0000 7f0bcd7c5700 1 -- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 msgr2=0x7f0bb4038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.55:6800/797433438 2023-12-19T19:07:04.291 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.289+0000 7f0bcd7c5700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 0x7f0bb4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-19T19:07:04.351 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.349+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0bc810e9e0 con 0x7f0bc80fab30 2023-12-19T19:07:04.351 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.349+0000 7f0bbe7fc700 1 -- 172.21.15.187:0/2974793105 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0bc4011370 con 0x7f0bc80fab30 2023-12-19T19:07:04.352 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:04.352 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 msgr2=0x7f0bb4038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 0x7f0bb4038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 msgr2=0x7f0bc810cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009040 tx=0x7f0bc4009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 shutdown_connections 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:6800/797433438,v1:172.21.15.55:6801/797433438] conn(0x7f0bb4036720 0x7f0bb4038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 --2- 172.21.15.187:0/2974793105 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f0bc80fab30 0x7f0bc810cab0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 >> 172.21.15.187:0/2974793105 conn(0x7f0bc80f65d0 msgr2=0x7f0bc80f7420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.351+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 shutdown_connections 2023-12-19T19:07:04.354 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:04.352+0000 7f0bd022a700 1 -- 172.21.15.187:0/2974793105 wait complete. 2023-12-19T19:07:04.355 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:04.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:04 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2974793105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:05.714 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: Active manager daemon smithi055.dwclvp restarted 2023-12-19T19:07:05.714 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: Activating manager daemon smithi055.dwclvp 2023-12-19T19:07:05.714 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T19:07:05.714 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: mgrmap e15: smithi055.dwclvp(active, starting, since 0.00733301s) 2023-12-19T19:07:05.714 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi055.dwclvp", "id": "smithi055.dwclvp"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: Manager daemon smithi055.dwclvp is now available 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/mirror_snapshot_schedule"}]: dispatch 2023-12-19T19:07:05.715 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/trash_purge_schedule"}]: dispatch 2023-12-19T19:07:05.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:05.939 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:06.186 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:07:06.234 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: [19/Dec/2023:19:07:05] ENGINE Bus STARTING 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: [19/Dec/2023:19:07:05] ENGINE Serving on https://172.21.15.55:7150 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: [19/Dec/2023:19:07:05] ENGINE Bus STARTED 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:06.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:06 smithi055 ceph-mon[80978]: mgrmap e16: smithi055.dwclvp(active, since 1.0053s) 2023-12-19T19:07:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:07 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 -- 172.21.15.187:0/3132023622 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 msgr2=0x7f53a00fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 --2- 172.21.15.187:0/3132023622 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a00fcc80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f53900048c0 tx=0x7f5390009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 -- 172.21.15.187:0/3132023622 shutdown_connections 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 --2- 172.21.15.187:0/3132023622 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a00fcc80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 -- 172.21.15.187:0/3132023622 >> 172.21.15.187:0/3132023622 conn(0x7f53a00f7f50 msgr2=0x7f53a00fa360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 -- 172.21.15.187:0/3132023622 shutdown_connections 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 -- 172.21.15.187:0/3132023622 wait complete. 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.791+0000 7f53a6991700 1 Processor -- start 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f53a6991700 1 -- start start 2023-12-19T19:07:08.795 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f53a6991700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f53a6991700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a010d020 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:35142/0 (socket says 172.21.15.187:35142) 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539ffff700 1 -- 172.21.15.187:0/1100702015 learned_addr learned my addr 172.21.15.187:0/1100702015 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539ffff700 1 -- 172.21.15.187:0/1100702015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390005040 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539ffff700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5390009330 tx=0x7f5390009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.792+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53900096b0 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5390009810 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a010d220 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f539001c5b0 con 0x7f53a00fc870 2023-12-19T19:07:08.796 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a010dec0 con 0x7f53a00fc870 2023-12-19T19:07:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f5390013420 con 0x7f53a00fc870 2023-12-19T19:07:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.793+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a005f270 con 0x7f53a00fc870 2023-12-19T19:07:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.794+0000 7f539cff9700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 0x7f5388038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.794+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5390049ca0 con 0x7f53a00fc870 2023-12-19T19:07:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.794+0000 7f539f7fe700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 0x7f5388038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:08.798 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.794+0000 7f539f7fe700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 0x7f5388038cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5394009990 tx=0x7f5394006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:08.801 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:08.799+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f53900119c0 con 0x7f53a00fc870 2023-12-19T19:07:08.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:08 smithi055 ceph-mon[80978]: mgrmap e17: smithi055.dwclvp(active, since 2s) 2023-12-19T19:07:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.074+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53a010ea10 con 0x7f53a00fc870 2023-12-19T19:07:09.077 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.075+0000 7f539cff9700 1 -- 172.21.15.187:0/1100702015 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5390011370 con 0x7f53a00fc870 2023-12-19T19:07:09.077 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:09.077 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 msgr2=0x7f5388038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 0x7f5388038cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5394009990 tx=0x7f5394006e40 comp rx=0 tx=0).stop 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 msgr2=0x7f53a010cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5390009330 tx=0x7f5390009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 shutdown_connections 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5388036810 0x7f5388038cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 --2- 172.21.15.187:0/1100702015 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f53a00fc870 0x7f53a010cae0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 >> 172.21.15.187:0/1100702015 conn(0x7f53a00f7f50 msgr2=0x7f53a00f8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:09.079 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 shutdown_connections 2023-12-19T19:07:09.080 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:09.077+0000 7f53a6991700 1 -- 172.21.15.187:0/1100702015 wait complete. 2023-12-19T19:07:09.080 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/1100702015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:07:10.888 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:10.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:11.135 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:07:11.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi055:/etc/ceph/ceph.conf 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi187:/etc/ceph/ceph.conf 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:11.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:12.551 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- 172.21.15.187:0/3994280656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 msgr2=0x7fb5200651f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:12.551 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 --2- 172.21.15.187:0/3994280656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb5200651f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0048c0 tx=0x7fb51c009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:12.551 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- 172.21.15.187:0/3994280656 shutdown_connections 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 --2- 172.21.15.187:0/3994280656 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb5200651f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- 172.21.15.187:0/3994280656 >> 172.21.15.187:0/3994280656 conn(0x7fb5200715e0 msgr2=0x7fb5200719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- 172.21.15.187:0/3994280656 shutdown_connections 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- 172.21.15.187:0/3994280656 wait complete. 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 Processor -- start 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.548+0000 7fb527245700 1 -- start start 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb527245700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb527245700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb520115260 con 0x7fb520064e20 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb524fe1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb524fe1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:40446/0 (socket says 172.21.15.187:40446) 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb524fe1700 1 -- 172.21.15.187:0/3227792560 learned_addr learned my addr 172.21.15.187:0/3227792560 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb524fe1700 1 -- 172.21.15.187:0/3227792560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c005040 con 0x7fb520064e20 2023-12-19T19:07:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.549+0000 7fb524fe1700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009040 tx=0x7fb51c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:12.553 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.551+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb51c004e60 con 0x7fb520064e20 2023-12-19T19:07:12.553 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.551+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb520115460 con 0x7fb520064e20 2023-12-19T19:07:12.553 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.551+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb51c009650 con 0x7fb520064e20 2023-12-19T19:07:12.553 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.551+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb51c01c3f0 con 0x7fb520064e20 2023-12-19T19:07:12.553 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.551+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb520115950 con 0x7fb520064e20 2023-12-19T19:07:12.554 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.552+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fb51c013420 con 0x7fb520064e20 2023-12-19T19:07:12.554 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.552+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52005f270 con 0x7fb520064e20 2023-12-19T19:07:12.554 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.552+0000 7fb5157fa700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 0x7fb508038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:12.554 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.552+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb51c049a90 con 0x7fb520064e20 2023-12-19T19:07:12.554 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.552+0000 7fb517fff700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 0x7fb508038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:12.555 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.553+0000 7fb517fff700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 0x7fb508038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb510009990 tx=0x7fb510006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:12.559 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.557+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb51c011550 con 0x7fb520064e20 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:12.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:07:12.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T19:07:12.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:12.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:12 smithi055 ceph-mon[80978]: Deploying daemon crash.smithi187 on smithi187 2023-12-19T19:07:12.829 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.826+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb520110490 con 0x7fb520064e20 2023-12-19T19:07:12.829 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.827+0000 7fb5157fa700 1 -- 172.21.15.187:0/3227792560 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb51c011370 con 0x7fb520064e20 2023-12-19T19:07:12.830 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:12.830 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 msgr2=0x7fb508038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 0x7fb508038c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb510009990 tx=0x7fb510006e40 comp rx=0 tx=0).stop 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 msgr2=0x7fb520114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb51c009040 tx=0x7fb51c009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 shutdown_connections 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5080367c0 0x7fb508038c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 --2- 172.21.15.187:0/3227792560 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb520064e20 0x7fb520114d20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 >> 172.21.15.187:0/3227792560 conn(0x7fb5200715e0 msgr2=0x7fb5200fa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 shutdown_connections 2023-12-19T19:07:12.832 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:12.829+0000 7fb527245700 1 -- 172.21.15.187:0/3227792560 wait complete. 2023-12-19T19:07:12.833 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:13 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/3227792560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:15.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:15.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:15.624 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:16.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:16 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:16.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:16 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:16.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:16 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:16.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:16 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:16.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:16 smithi055 ceph-mon[80978]: Deploying daemon node-exporter.smithi187 on smithi187 2023-12-19T19:07:17.025 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.022+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/1804129555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 msgr2=0x7fd2d00fdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.022+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/1804129555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d00fdc50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80048c0 tx=0x7fd2c8009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/1804129555 shutdown_connections 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/1804129555 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d00fdc50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/1804129555 >> 172.21.15.187:0/1804129555 conn(0x7fd2d00f7310 msgr2=0x7fd2d00f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/1804129555 shutdown_connections 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/1804129555 wait complete. 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 Processor -- start 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- start start 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2d4ae9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d0113110 con 0x7fd2d00fd880 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:17.026 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2ce59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:40480/0 (socket says 172.21.15.187:40480) 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.023+0000 7fd2ce59c700 1 -- 172.21.15.187:0/2904889322 learned_addr learned my addr 172.21.15.187:0/2904889322 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2ce59c700 1 -- 172.21.15.187:0/2904889322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8005040 con 0x7fd2d00fd880 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2ce59c700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009300 tx=0x7fd2c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2c8004e60 con 0x7fd2d00fd880 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d0113310 con 0x7fd2d00fd880 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2c8013420 con 0x7fd2d00fd880 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2c801c5b0 con 0x7fd2d00fd880 2023-12-19T19:07:17.027 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.024+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d0113780 con 0x7fd2d00fd880 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.025+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fd2c80139e0 con 0x7fd2d00fd880 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.025+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d005f270 con 0x7fd2d00fd880 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.025+0000 7fd2beffd700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 0x7fd2b8038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.025+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd2c8049ab0 con 0x7fd2d00fd880 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.025+0000 7fd2cdd9b700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 0x7fd2b8038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.026+0000 7fd2cdd9b700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 0x7fd2b8038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400ad90 tx=0x7fd2c4009410 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:17.032 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.031+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd2c8011960 con 0x7fd2d00fd880 2023-12-19T19:07:17.331 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.329+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd2d004bc10 con 0x7fd2d00fd880 2023-12-19T19:07:17.332 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.330+0000 7fd2beffd700 1 -- 172.21.15.187:0/2904889322 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd2c8011370 con 0x7fd2d00fd880 2023-12-19T19:07:17.332 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:17.332 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 msgr2=0x7fd2b8038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 0x7fd2b8038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400ad90 tx=0x7fd2c4009410 comp rx=0 tx=0).stop 2023-12-19T19:07:17.334 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 msgr2=0x7fd2d0112bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009300 tx=0x7fd2c8009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 shutdown_connections 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd2b80367c0 0x7fd2b8038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 --2- 172.21.15.187:0/2904889322 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd2d00fd880 0x7fd2d0112bd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 >> 172.21.15.187:0/2904889322 conn(0x7fd2d00f7310 msgr2=0x7fd2d01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 shutdown_connections 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:17.332+0000 7fd2d4ae9700 1 -- 172.21.15.187:0/2904889322 wait complete. 2023-12-19T19:07:17.335 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:17 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2904889322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:18.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:18.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:19.381 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:21.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- 172.21.15.187:0/358747547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 msgr2=0x7fb818102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:21.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/358747547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818102cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb8080048c0 tx=0x7fb808009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- 172.21.15.187:0/358747547 shutdown_connections 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/358747547 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818102cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- 172.21.15.187:0/358747547 >> 172.21.15.187:0/358747547 conn(0x7fb8180715e0 msgr2=0x7fb8180719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- 172.21.15.187:0/358747547 shutdown_connections 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- 172.21.15.187:0/358747547 wait complete. 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 Processor -- start 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.082+0000 7fb81c8f5700 1 -- start start 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81c8f5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81c8f5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb818113090 con 0x7fb8181028f0 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:55506/0 (socket says 172.21.15.187:55506) 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81659c700 1 -- 172.21.15.187:0/2703978956 learned_addr learned my addr 172.21.15.187:0/2703978956 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81659c700 1 -- 172.21.15.187:0/2703978956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808005040 con 0x7fb8181028f0 2023-12-19T19:07:21.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81659c700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb808009040 tx=0x7fb808009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb808004e60 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb808009650 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb818113290 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb80801c3f0 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.083+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81810ca10 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.084+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81805f270 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.084+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fb8080097c0 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.084+0000 7fb806ffd700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 0x7fb800038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.085+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb8080499c0 con 0x7fb8181028f0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.085+0000 7fb815d9b700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 0x7fb800038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.085+0000 7fb815d9b700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 0x7fb800038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009990 tx=0x7fb80c006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:21.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.090+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb808011940 con 0x7fb8181028f0 2023-12-19T19:07:21.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:21.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.367+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb81810e130 con 0x7fb8181028f0 2023-12-19T19:07:21.369 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.367+0000 7fb806ffd700 1 -- 172.21.15.187:0/2703978956 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb808011370 con 0x7fb8181028f0 2023-12-19T19:07:21.370 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:21.370 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 msgr2=0x7fb800038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 0x7fb800038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009990 tx=0x7fb80c006e40 comp rx=0 tx=0).stop 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 msgr2=0x7fb818112b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb808009040 tx=0x7fb808009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 shutdown_connections 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb8000367c0 0x7fb800038c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 --2- 172.21.15.187:0/2703978956 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb8181028f0 0x7fb818112b50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 >> 172.21.15.187:0/2703978956 conn(0x7fb8180715e0 msgr2=0x7fb818103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 shutdown_connections 2023-12-19T19:07:21.372 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:21.369+0000 7fb81c8f5700 1 -- 172.21.15.187:0/2703978956 wait complete. 2023-12-19T19:07:21.373 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:21 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2703978956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:23.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:23.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:24.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:07:24.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-19T19:07:24.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:07:24.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:24.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:23 smithi055 ceph-mon[80978]: Deploying daemon mgr.smithi187.pmhvpw on smithi187 2023-12-19T19:07:24.427 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 -- 172.21.15.187:0/2750134078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 msgr2=0x7fca70102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 --2- 172.21.15.187:0/2750134078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70102c80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0048c0 tx=0x7fca6c009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 -- 172.21.15.187:0/2750134078 shutdown_connections 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 --2- 172.21.15.187:0/2750134078 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70102c80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 -- 172.21.15.187:0/2750134078 >> 172.21.15.187:0/2750134078 conn(0x7fca700f8410 msgr2=0x7fca700fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 -- 172.21.15.187:0/2750134078 shutdown_connections 2023-12-19T19:07:26.019 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.988+0000 7fca77930700 1 -- 172.21.15.187:0/2750134078 wait complete. 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca77930700 1 Processor -- start 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca77930700 1 -- start start 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca77930700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca77930700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca701130a0 con 0x7fca701028b0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca756cc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca756cc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:55518/0 (socket says 172.21.15.187:55518) 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.989+0000 7fca756cc700 1 -- 172.21.15.187:0/3985549826 learned_addr learned my addr 172.21.15.187:0/3985549826 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca756cc700 1 -- 172.21.15.187:0/3985549826 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c005040 con 0x7fca701028b0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca756cc700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009300 tx=0x7fca6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca6c004e60 con 0x7fca701028b0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca6c013420 con 0x7fca701028b0 2023-12-19T19:07:26.020 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca701132a0 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca6c01c5b0 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.990+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7010ca20 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.991+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7005f270 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.991+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7fca6c0139e0 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.991+0000 7fca65ffb700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 0x7fca5c038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.991+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fca6c049a20 con 0x7fca701028b0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.992+0000 7fca74ecb700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 0x7fca5c038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.992+0000 7fca74ecb700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 0x7fca5c038cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fca60009990 tx=0x7fca60006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:26.021 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:25.997+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fca6c011960 con 0x7fca701028b0 2023-12-19T19:07:26.265 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.262+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca7004bbe0 con 0x7fca701028b0 2023-12-19T19:07:26.268 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.266+0000 7fca65ffb700 1 -- 172.21.15.187:0/3985549826 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fca6c011370 con 0x7fca701028b0 2023-12-19T19:07:26.268 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:26.268 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:26.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 msgr2=0x7fca5c038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:26.270 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 0x7fca5c038cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fca60009990 tx=0x7fca60006e40 comp rx=0 tx=0).stop 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 msgr2=0x7fca70112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009300 tx=0x7fca6c009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 shutdown_connections 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca5c036810 0x7fca5c038cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 --2- 172.21.15.187:0/3985549826 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca701028b0 0x7fca70112b60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 >> 172.21.15.187:0/3985549826 conn(0x7fca700f8410 msgr2=0x7fca700f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 shutdown_connections 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:26.268+0000 7fca77930700 1 -- 172.21.15.187:0/3985549826 wait complete. 2023-12-19T19:07:26.271 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:26.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:25 smithi055 ceph-mon[80978]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:26.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:27.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/3985549826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:27.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:27.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:27.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:27.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:27.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:07:27.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:28.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:28.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:28.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:27 smithi055 ceph-mon[80978]: Deploying daemon mon.smithi187 on smithi187 2023-12-19T19:07:28.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:27 smithi055 ceph-mon[80978]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:29.352 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:07:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:29 smithi055 ceph-mon[80978]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.046 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 -- 172.21.15.187:0/2447360798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 msgr2=0x7f73640fddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 --2- 172.21.15.187:0/2447360798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f73640fddb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f73540048c0 tx=0x7f7354009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 -- 172.21.15.187:0/2447360798 shutdown_connections 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 --2- 172.21.15.187:0/2447360798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f73640fddb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 -- 172.21.15.187:0/2447360798 >> 172.21.15.187:0/2447360798 conn(0x7f73640f7410 msgr2=0x7f73640f9820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 -- 172.21.15.187:0/2447360798 shutdown_connections 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.044+0000 7f7369d3c700 1 -- 172.21.15.187:0/2447360798 wait complete. 2023-12-19T19:07:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.045+0000 7f7369d3c700 1 Processor -- start 2023-12-19T19:07:31.048 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.045+0000 7f7369d3c700 1 -- start start 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.045+0000 7f7369d3c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.045+0000 7f7369d3c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7364113260 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f73637fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f73637fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.187:33356/0 (socket says 172.21.15.187:33356) 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f73637fe700 1 -- 172.21.15.187:0/4270539868 learned_addr learned my addr 172.21.15.187:0/4270539868 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f73637fe700 1 -- 172.21.15.187:0/4270539868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354005040 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f73637fe700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7354009300 tx=0x7f7354009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7354004e60 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7354013420 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f735401c5b0 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.046+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7364113460 con 0x7f73640fd9e0 2023-12-19T19:07:31.049 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.047+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7364113810 con 0x7f73640fd9e0 2023-12-19T19:07:31.050 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.047+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f735401c7f0 con 0x7f73640fd9e0 2023-12-19T19:07:31.050 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.048+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736405f270 con 0x7f73640fd9e0 2023-12-19T19:07:31.050 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.048+0000 7f734bfff700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 0x7f734c038c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:31.050 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.048+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7354049a20 con 0x7f73640fd9e0 2023-12-19T19:07:31.050 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.048+0000 7f7362ffd700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 0x7f734c038c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:31.051 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.048+0000 7f7362ffd700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 0x7f734c038c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7358009990 tx=0x7f7358006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:31.055 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.053+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7354025400 con 0x7f73640fd9e0 2023-12-19T19:07:31.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:30 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.338 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.335+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f736410e470 con 0x7f73640fd9e0 2023-12-19T19:07:31.339 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.336+0000 7f734bfff700 1 -- 172.21.15.187:0/4270539868 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7354011b60 con 0x7f73640fd9e0 2023-12-19T19:07:31.339 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:31.339 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:04:55.017268Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.338+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 msgr2=0x7f734c038c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.338+0000 7f7369d3c700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 0x7f734c038c80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7358009990 tx=0x7f7358006e40 comp rx=0 tx=0).stop 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.338+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 msgr2=0x7f7364112d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.338+0000 7f7369d3c700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7354009300 tx=0x7f7354009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 shutdown_connections 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f734c0367d0 0x7f734c038c80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 --2- 172.21.15.187:0/4270539868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f73640fd9e0 0x7f7364112d20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:31.341 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 >> 172.21.15.187:0/4270539868 conn(0x7f73640f7410 msgr2=0x7f7364101200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:31.345 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 shutdown_connections 2023-12-19T19:07:31.345 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:31.339+0000 7f7369d3c700 1 -- 172.21.15.187:0/4270539868 wait complete. 2023-12-19T19:07:31.345 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2023-12-19T19:07:31.504 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Added host smithi187 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon grafana.smithi055 on smithi055 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1756681708' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1756681708' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: osdmap e4: 0 total, 0 up, 0 in 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Saving service mon spec with placement smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187;count:2 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/3617469307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/187273837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/749057785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon prometheus.smithi055 on smithi055 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.843 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2797835074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/1547484737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2189299774' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14162 172.21.15.55:0/1580559350' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-19T19:07:31.844 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mgrmap e14: smithi055.dwclvp(active, since 55s) 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/3250985082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2974793105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Active manager daemon smithi055.dwclvp restarted 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Activating manager daemon smithi055.dwclvp 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mgrmap e15: smithi055.dwclvp(active, starting, since 0.00733301s) 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:07:31.845 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi055.dwclvp", "id": "smithi055.dwclvp"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Manager daemon smithi055.dwclvp is now available 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:31.847 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/mirror_snapshot_schedule"}]: dispatch 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi055.dwclvp/trash_purge_schedule"}]: dispatch 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: [19/Dec/2023:19:07:05] ENGINE Bus STARTING 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: [19/Dec/2023:19:07:05] ENGINE Serving on https://172.21.15.55:7150 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: [19/Dec/2023:19:07:05] ENGINE Bus STARTED 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mgrmap e16: smithi055.dwclvp(active, since 1.0053s) 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mgrmap e17: smithi055.dwclvp(active, since 2s) 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/1100702015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.848 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi055", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi055:/etc/ceph/ceph.conf 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi187:/etc/ceph/ceph.conf 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:31.849 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi055:/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon crash.smithi187 on smithi187 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/3227792560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.850 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon node-exporter.smithi187 on smithi187 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2904889322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2703978956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:31.851 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon mgr.smithi187.pmhvpw on smithi187 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/3985549826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: Deploying daemon mon.smithi187 on smithi187 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mon.smithi187@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 ceph-mon[87698]: mon.smithi187@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-19T19:07:31.852 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 bash[87663]: 31c58c5214bc806d165015cbd4f559913eac95f0df9e7c3d1efe92023cef9262 2023-12-19T19:07:31.853 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:31 smithi187 systemd[1]: Started Ceph mon.smithi187 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:07:33.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-19T19:07:33.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-19T19:07:33.541 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:07:37.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:07:37.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: mon.smithi055 calling monitor election 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.pmhvpw/crt"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: mon.smithi187 calling monitor election 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: mon.smithi055 is new leader, mons smithi055,smithi187 in quorum (ranks 0,1) 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: monmap e2: 2 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],smithi187=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T19:07:37.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: fsmap 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: mgrmap e17: smithi055.dwclvp(active, since 31s) 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: Standby manager daemon smithi187.pmhvpw started 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: overall HEALTH_OK 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.pmhvpw/key"}]: dispatch 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:36 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: mon.smithi055 calling monitor election 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.pmhvpw/crt"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: mon.smithi187 calling monitor election 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: mon.smithi055 is new leader, mons smithi055,smithi187 in quorum (ranks 0,1) 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: monmap e2: 2 mons at {smithi055=[v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0],smithi187=[v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-19T19:07:37.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: fsmap 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: osdmap e5: 0 total, 0 up, 0 in 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: mgrmap e17: smithi055.dwclvp(active, since 31s) 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: Standby manager daemon smithi187.pmhvpw started 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: overall HEALTH_OK 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi187.pmhvpw/key"}]: dispatch 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.? 172.21.15.187:0/3716385426' entity='mgr.smithi187.pmhvpw' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:36 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.983 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: mgrmap e18: smithi055.dwclvp(active, since 31s), standbys: smithi187.pmhvpw 2023-12-19T19:07:37.983 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:37.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi187.pmhvpw", "id": "smithi187.pmhvpw"}]: dispatch 2023-12-19T19:07:37.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:37.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:37.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:37 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: mgrmap e18: smithi055.dwclvp(active, since 31s), standbys: smithi187.pmhvpw 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr metadata", "who": "smithi187.pmhvpw", "id": "smithi187.pmhvpw"}]: dispatch 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:38.013 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:37 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/2754540005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 msgr2=0x7f1c60005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/2754540005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c60005a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1c740fb7c0 tx=0x7f1c7000b040 comp rx=0 tx=0).stop 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/2754540005 shutdown_connections 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/2754540005 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c60005a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/2754540005 >> 172.21.15.187:0/2754540005 conn(0x7f1c740f73f0 msgr2=0x7f1c740f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:38.102 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/2754540005 shutdown_connections 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.097+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/2754540005 wait complete. 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 Processor -- start 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 -- start start 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c74111c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74112700 con 0x7f1c60003580 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7b6c1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7410c000 con 0x7f1c74112140 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c7945d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c74111c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:38.103 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49944/0 (socket says 172.21.15.187:49944) 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 -- 172.21.15.187:0/1380947104 learned_addr learned my addr 172.21.15.187:0/1380947104 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 -- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 msgr2=0x7f1c7410bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 -- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 msgr2=0x7f1c7410bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.098+0000 7f1c78c5c700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7945d700 1 -- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 msgr2=0x7f1c7410bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7945d700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7945d700 1 -- 172.21.15.187:0/1380947104 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c70005040 con 0x7f1c60003580 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7945d700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c74111c00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000b010 tx=0x7f1c7000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c7001b070 con 0x7f1c60003580 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7410c280 con 0x7f1c60003580 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c700208f0 con 0x7f1c60003580 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c7001f850 con 0x7f1c60003580 2023-12-19T19:07:38.104 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.099+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7410c790 con 0x7f1c60003580 2023-12-19T19:07:38.105 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.100+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7405f270 con 0x7f1c60003580 2023-12-19T19:07:38.105 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.101+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f1c70020a60 con 0x7f1c60003580 2023-12-19T19:07:38.105 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.101+0000 7f1c69ffb700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 0x7f1c5806b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:38.105 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.101+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1c70087fa0 con 0x7f1c60003580 2023-12-19T19:07:38.106 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.101+0000 7f1c78c5c700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 0x7f1c5806b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:38.106 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.102+0000 7f1c78c5c700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 0x7f1c5806b4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c64006fd0 tx=0x7f1c64008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:38.113 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.110+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1c70058f40 con 0x7f1c60003580 2023-12-19T19:07:38.377 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.375+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c7410d1e0 con 0x7f1c60003580 2023-12-19T19:07:38.378 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.375+0000 7f1c69ffb700 1 -- 172.21.15.187:0/1380947104 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f1c70024020 con 0x7f1c60003580 2023-12-19T19:07:38.378 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-12-19T19:07:38.378 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","modified":"2023-12-19T19:07:31.509949Z","created":"2023-12-19T19:04:55.017268Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi055","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:3300","nonce":0},{"type":"v1","addr":"172.21.15.55:6789","nonce":0}]},"addr":"172.21.15.55:6789/0","public_addr":"172.21.15.55:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi187","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 msgr2=0x7f1c5806b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 0x7f1c5806b4d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1c64006fd0 tx=0x7f1c64008040 comp rx=0 tx=0).stop 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 msgr2=0x7f1c74111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c74111c00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000b010 tx=0x7f1c7000b040 comp rx=0 tx=0).stop 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 shutdown_connections 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1c58069020 0x7f1c5806b4d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1c60003580 0x7f1c74111c00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 --2- 172.21.15.187:0/1380947104 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1c74112140 0x7f1c7410bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 >> 172.21.15.187:0/1380947104 conn(0x7f1c740f73f0 msgr2=0x7f1c740fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 shutdown_connections 2023-12-19T19:07:38.380 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:07:38.378+0000 7f1c7b6c1700 1 -- 172.21.15.187:0/1380947104 wait complete. 2023-12-19T19:07:38.381 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2023-12-19T19:07:38.735 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:38 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/1380947104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:38.932 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:38 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/1380947104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-19T19:07:39.042 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-19T19:07:39.042 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-19T19:07:39.336 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:07:39.739 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:39.740 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Updating smithi055:/etc/ceph/ceph.conf 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Updating smithi187:/etc/ceph/ceph.conf 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Reconfiguring mon.smithi055 (unknown last config time)... 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T19:07:40.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:40.916 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: Reconfiguring daemon mon.smithi055 on smithi055 2023-12-19T19:07:40.916 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Updating smithi055:/etc/ceph/ceph.conf 2023-12-19T19:07:41.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Updating smithi187:/etc/ceph/ceph.conf 2023-12-19T19:07:41.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Updating smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:41.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Updating smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Reconfiguring mon.smithi055 (unknown last config time)... 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T19:07:41.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:41.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: Reconfiguring daemon mon.smithi055 on smithi055 2023-12-19T19:07:41.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 -- 172.21.15.55:0/1529790232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca44102c70 msgr2=0x7fca44103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 --2- 172.21.15.55:0/1529790232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca44102c70 0x7fca44103040 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fca34004940 tx=0x7fca34009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 -- 172.21.15.55:0/1529790232 shutdown_connections 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 --2- 172.21.15.55:0/1529790232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca440fb1e0 0x7fca440fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 --2- 172.21.15.55:0/1529790232 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca44102c70 0x7fca44103040 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 -- 172.21.15.55:0/1529790232 >> 172.21.15.55:0/1529790232 conn(0x7fca440711a0 msgr2=0x7fca440715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 -- 172.21.15.55:0/1529790232 shutdown_connections 2023-12-19T19:07:41.608 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.602+0000 7fca489f5700 1 -- 172.21.15.55:0/1529790232 wait complete. 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 Processor -- start 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 -- start start 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 0x7fca440fc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440fcf90 con 0x7fca440fb1e0 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca489f5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440fd100 con 0x7fca44102c70 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca4259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56994/0 (socket says 172.21.15.55:56994) 2023-12-19T19:07:41.609 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.603+0000 7fca4259c700 1 -- 172.21.15.55:0/779854442 learned_addr learned my addr 172.21.15.55:0/779854442 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca41d9b700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 0x7fca440fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca4259c700 1 -- 172.21.15.55:0/779854442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 msgr2=0x7fca440fc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca4259c700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 0x7fca440fc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca4259c700 1 -- 172.21.15.55:0/779854442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34005040 con 0x7fca440fb1e0 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca41d9b700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 0x7fca440fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca4259c700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fca3400b3b0 tx=0x7fca34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca3400e040 con 0x7fca440fb1e0 2023-12-19T19:07:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca3400bc50 con 0x7fca440fb1e0 2023-12-19T19:07:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca44119830 con 0x7fca440fb1e0 2023-12-19T19:07:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca34013420 con 0x7fca440fb1e0 2023-12-19T19:07:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.604+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca44119d20 con 0x7fca440fb1e0 2023-12-19T19:07:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.606+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fca3401c430 con 0x7fca440fb1e0 2023-12-19T19:07:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.606+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca44072cc0 con 0x7fca440fb1e0 2023-12-19T19:07:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.606+0000 7fca32ffd700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 0x7fca2c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.606+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fca34086a60 con 0x7fca440fb1e0 2023-12-19T19:07:41.613 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.606+0000 7fca41d9b700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 0x7fca2c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:41.613 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.607+0000 7fca41d9b700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 0x7fca2c06b380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca38006fd0 tx=0x7fca38008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:41.613 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.611+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fca34057b50 con 0x7fca440fb1e0 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi055.dwclvp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:07:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:41.812 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.810+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fca440fdb00 con 0x7fca440fb1e0 2023-12-19T19:07:41.812 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.810+0000 7fca32ffd700 1 -- 172.21.15.55:0/779854442 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fca34086210 con 0x7fca440fb1e0 2023-12-19T19:07:41.813 INFO:teuthology.orchestra.run.smithi055.stdout:# minimal ceph.conf for 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:07:41.813 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2023-12-19T19:07:41.813 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 63de9ba6-9ea1-11ee-95a5-87774f69a715 2023-12-19T19:07:41.813 INFO:teuthology.orchestra.run.smithi055.stdout: mon_host = [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2023-12-19T19:07:41.815 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.812+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 msgr2=0x7fca2c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:41.815 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.812+0000 7fca489f5700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 0x7fca2c06b380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca38006fd0 tx=0x7fca38008040 comp rx=0 tx=0).stop 2023-12-19T19:07:41.815 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.812+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 msgr2=0x7fca440fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.812+0000 7fca489f5700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fca3400b3b0 tx=0x7fca34009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 shutdown_connections 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fca2c068ed0 0x7fca2c06b380 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fca440fb1e0 0x7fca440fc3c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 --2- 172.21.15.55:0/779854442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fca44102c70 0x7fca440fc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 >> 172.21.15.55:0/779854442 conn(0x7fca440711a0 msgr2=0x7fca440fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 shutdown_connections 2023-12-19T19:07:41.816 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:41.813+0000 7fca489f5700 1 -- 172.21.15.55:0/779854442 wait complete. 2023-12-19T19:07:41.991 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:41.992 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.992 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:41.992 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi055.dwclvp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:07:41.992 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:07:41.992 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:42.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:42 smithi055 ceph-mon[80978]: Reconfiguring mgr.smithi055.dwclvp (unknown last config time)... 2023-12-19T19:07:42.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:42 smithi055 ceph-mon[80978]: Reconfiguring daemon mgr.smithi055.dwclvp on smithi055 2023-12-19T19:07:42.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:42 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/779854442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:43.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:42 smithi187 ceph-mon[87698]: Reconfiguring mgr.smithi055.dwclvp (unknown last config time)... 2023-12-19T19:07:43.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:42 smithi187 ceph-mon[87698]: Reconfiguring daemon mgr.smithi055.dwclvp on smithi055 2023-12-19T19:07:43.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:42 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/779854442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:43.391 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-19T19:07:43.392 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:07:43.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.conf 2023-12-19T19:07:43.427 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:07:43.427 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:43.499 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:07:43.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2023-12-19T19:07:43.529 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:07:43.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-19T19:07:43.597 INFO:tasks.cephadm:Deploying OSDs... 2023-12-19T19:07:43.597 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:07:43.597 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T19:07:43.614 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T19:07:43.614 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2023-12-19T19:07:43.669 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-19 19:06:56.257199183 +0000 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-19 19:00:39.089509564 +0000 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-19 19:00:39.089509564 +0000 2023-12-19T19:07:43.670 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-12-19 19:00:39.089509564 +0000 2023-12-19T19:07:43.671 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T19:07:43.739 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-19T19:07:43.740 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-19T19:07:43.740 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000272268 s, 1.9 MB/s 2023-12-19T19:07:43.741 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T19:07:43.798 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2023-12-19T19:07:43.855 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T19:07:43.855 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:43.855 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-19T19:07:43.855 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:43.856 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:43.856 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-19 19:06:56.258199164 +0000 2023-12-19T19:07:43.856 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-19 19:00:39.610499387 +0000 2023-12-19T19:07:43.856 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-19 19:00:39.610499387 +0000 2023-12-19T19:07:43.856 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-12-19 19:00:39.610499387 +0000 2023-12-19T19:07:43.857 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T19:07:43.924 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-19T19:07:43.924 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-19T19:07:43.924 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.00026829 s, 1.9 MB/s 2023-12-19T19:07:43.925 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: Reconfiguring crash.smithi055 (monmap changed)... 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:07:43.970 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:43.971 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:43 smithi055 ceph-mon[80978]: Reconfiguring daemon crash.smithi055 on smithi055 2023-12-19T19:07:43.982 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2023-12-19T19:07:44.037 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-19 19:06:56.258199164 +0000 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-19 19:00:40.110489620 +0000 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-19 19:00:40.110489620 +0000 2023-12-19T19:07:44.038 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-12-19 19:00:40.110489620 +0000 2023-12-19T19:07:44.038 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T19:07:44.107 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-19T19:07:44.107 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-19T19:07:44.107 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.00027632 s, 1.9 MB/s 2023-12-19T19:07:44.109 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T19:07:44.165 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-19 19:06:56.259199144 +0000 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-19 19:00:40.610479853 +0000 2023-12-19T19:07:44.221 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-19 19:00:40.610479853 +0000 2023-12-19T19:07:44.222 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-12-19 19:00:40.610479853 +0000 2023-12-19T19:07:44.222 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T19:07:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-19T19:07:44.291 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-19T19:07:44.291 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000275974 s, 1.9 MB/s 2023-12-19T19:07:44.292 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: Reconfiguring crash.smithi055 (monmap changed)... 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi055", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:43 smithi187 ceph-mon[87698]: Reconfiguring daemon crash.smithi055 on smithi055 2023-12-19T19:07:44.349 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:07:44.350 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T19:07:44.367 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T19:07:44.367 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2023-12-19T19:07:44.424 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-12-19 19:07:29.625806410 +0000 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-12-19 19:00:35.007797674 +0000 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-12-19 19:00:35.007797674 +0000 2023-12-19T19:07:44.425 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2023-12-19 19:00:35.007797674 +0000 2023-12-19T19:07:44.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T19:07:44.492 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-12-19T19:07:44.492 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-12-19T19:07:44.493 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000222913 s, 2.3 MB/s 2023-12-19T19:07:44.494 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T19:07:44.551 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2023-12-19T19:07:44.608 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T19:07:44.608 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.608 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-19T19:07:44.608 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.608 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.609 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-12-19 19:07:29.625806410 +0000 2023-12-19T19:07:44.609 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-12-19 19:00:35.499789996 +0000 2023-12-19T19:07:44.609 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-12-19 19:00:35.499789996 +0000 2023-12-19T19:07:44.609 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2023-12-19 19:00:35.499789996 +0000 2023-12-19T19:07:44.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T19:07:44.675 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-12-19T19:07:44.675 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-12-19T19:07:44.676 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000222883 s, 2.3 MB/s 2023-12-19T19:07:44.676 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T19:07:44.734 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.789 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-12-19 19:07:29.626806394 +0000 2023-12-19T19:07:44.790 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-12-19 19:00:36.004782115 +0000 2023-12-19T19:07:44.790 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-12-19 19:00:36.004782115 +0000 2023-12-19T19:07:44.790 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2023-12-19 19:00:36.004782115 +0000 2023-12-19T19:07:44.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T19:07:44.854 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-12-19T19:07:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-12-19T19:07:44.855 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000220124 s, 2.3 MB/s 2023-12-19T19:07:44.856 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T19:07:44.911 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2023-12-19 19:07:29.626806394 +0000 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-12-19 19:00:36.501774358 +0000 2023-12-19T19:07:44.967 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-12-19 19:00:36.501774358 +0000 2023-12-19T19:07:44.968 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2023-12-19 19:00:36.501774358 +0000 2023-12-19T19:07:44.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T19:07:45.033 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2023-12-19T19:07:45.033 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2023-12-19T19:07:45.033 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000238196 s, 2.1 MB/s 2023-12-19T19:07:45.034 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T19:07:45.091 INFO:tasks.cephadm:Deploying osd.0 on smithi055 with /dev/vg_nvme/lv_4... 2023-12-19T19:07:45.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-19T19:07:45.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:45 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:45.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:45 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:45.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:45 smithi187 ceph-mon[87698]: Reconfiguring alertmanager.smithi055 (dependencies changed)... 2023-12-19T19:07:45.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:45 smithi187 ceph-mon[87698]: Reconfiguring daemon alertmanager.smithi055 on smithi055 2023-12-19T19:07:45.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:45 smithi187 ceph-mon[87698]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:45.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:45 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:45.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:45 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:45.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:45 smithi055 ceph-mon[80978]: Reconfiguring alertmanager.smithi055 (dependencies changed)... 2023-12-19T19:07:45.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:45 smithi055 ceph-mon[80978]: Reconfiguring daemon alertmanager.smithi055 on smithi055 2023-12-19T19:07:45.959 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:45 smithi055 ceph-mon[80978]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:47.330 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:07:47.993 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:47 smithi055 ceph-mon[80978]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:48.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:47 smithi187 ceph-mon[87698]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116646 s, 899 MB/s 2023-12-19T19:07:49.966 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2023-12-19T19:07:50.121 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:49 smithi055 ceph-mon[80978]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:50.121 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:49 smithi187 ceph-mon[87698]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:07:51.137 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_4 2023-12-19T19:07:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:51 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:51 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:51 smithi055 ceph-mon[80978]: Reconfiguring grafana.smithi055 (dependencies changed)... 2023-12-19T19:07:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:51 smithi055 ceph-mon[80978]: Reconfiguring daemon grafana.smithi055 on smithi055 2023-12-19T19:07:51.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:51 smithi055 ceph-mon[80978]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:51 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:51 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:51 smithi187 ceph-mon[87698]: Reconfiguring grafana.smithi055 (dependencies changed)... 2023-12-19T19:07:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:51 smithi187 ceph-mon[87698]: Reconfiguring daemon grafana.smithi055 on smithi055 2023-12-19T19:07:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:51 smithi187 ceph-mon[87698]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:51.873 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:07:53.742 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:53 smithi055 ceph-mon[80978]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:53 smithi187 ceph-mon[87698]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:54.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 -- 172.21.15.55:0/493442496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1120100060 msgr2=0x7f1120102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:54.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 --2- 172.21.15.55:0/493442496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1120100060 0x7f1120102440 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1110004940 tx=0x7f1110009e30 comp rx=0 tx=0).stop 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 -- 172.21.15.55:0/493442496 shutdown_connections 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 --2- 172.21.15.55:0/493442496 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1120100060 0x7f1120102440 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 --2- 172.21.15.55:0/493442496 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11200fd740 0x7f11200ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 -- 172.21.15.55:0/493442496 >> 172.21.15.55:0/493442496 conn(0x7f11200f71c0 msgr2=0x7f11200f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 -- 172.21.15.55:0/493442496 shutdown_connections 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.534+0000 7f112838a700 1 -- 172.21.15.55:0/493442496 wait complete. 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 Processor -- start 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 -- start start 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 0x7f11200fb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11200fbc50 con 0x7f11200fd740 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f112838a700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11200fbdc0 con 0x7f1120100060 2023-12-19T19:07:54.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48200/0 (socket says 172.21.15.55:48200) 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 -- 172.21.15.55:0/3047700360 learned_addr learned my addr 172.21.15.55:0/3047700360 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1125925700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 0x7f11200fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 -- 172.21.15.55:0/3047700360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 msgr2=0x7f11200fb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 0x7f11200fb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 -- 172.21.15.55:0/3047700360 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1110005040 con 0x7f11200fd740 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1125925700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 0x7f11200fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.535+0000 7f1126126700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f111c002a50 tx=0x7f111c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.536+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111c007cc0 con 0x7f11200fd740 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.536+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f111c00f040 con 0x7f11200fd740 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.536+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11200fc0a0 con 0x7f11200fd740 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.536+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f111c009540 con 0x7f11200fd740 2023-12-19T19:07:54.540 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.536+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1120115570 con 0x7f11200fd740 2023-12-19T19:07:54.541 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.537+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11201075d0 con 0x7f11200fd740 2023-12-19T19:07:54.541 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.537+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f111c0096c0 con 0x7f11200fd740 2023-12-19T19:07:54.541 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.538+0000 7f1116ffd700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 0x7f110c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:07:54.541 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.538+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f111c0859b0 con 0x7f11200fd740 2023-12-19T19:07:54.542 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.538+0000 7f1125925700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 0x7f110c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:07:54.542 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.538+0000 7f1125925700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 0x7f110c06b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1110000c00 tx=0x7f1110009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:07:54.545 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.543+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f111c0569f0 con 0x7f11200fd740 2023-12-19T19:07:54.757 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:07:54.754+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f112002b150 con 0x7f110c068ed0 2023-12-19T19:07:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:54 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:07:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:54 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:07:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:54 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:55.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:54 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:07:55.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:54 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:07:55.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:54 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:07:56.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:55 smithi055 ceph-mon[80978]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:07:56.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:55 smithi055 ceph-mon[80978]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:56.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:55 smithi187 ceph-mon[87698]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:07:56.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:55 smithi187 ceph-mon[87698]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:57 smithi055 ceph-mon[80978]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:57 smithi187 ceph-mon[87698]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:07:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: Reconfiguring prometheus.smithi055 (dependencies changed)... 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: Reconfiguring daemon prometheus.smithi055 on smithi055 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3907511231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ca473b-4320-4248-b6b1-fb3182ca21ee"}]: dispatch 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3907511231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4ca473b-4320-4248-b6b1-fb3182ca21ee"}]': finished 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: osdmap e6: 1 total, 0 up, 1 in 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:07:59.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:07:59 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1289824997' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: Reconfiguring prometheus.smithi055 (dependencies changed)... 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: Reconfiguring daemon prometheus.smithi055 on smithi055 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3907511231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4ca473b-4320-4248-b6b1-fb3182ca21ee"}]: dispatch 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3907511231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4ca473b-4320-4248-b6b1-fb3182ca21ee"}]': finished 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: osdmap e6: 1 total, 0 up, 1 in 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:07:59.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:07:59 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1289824997' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:08:00.380 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:00 smithi055 ceph-mon[80978]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:00.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:00 smithi187 ceph-mon[87698]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:01.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:01 smithi055 ceph-mon[80978]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:01.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:01 smithi187 ceph-mon[87698]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:03.632 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:03 smithi055 ceph-mon[80978]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:03.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:03 smithi187 ceph-mon[87698]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:05.202 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:05.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:06.142 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:05 smithi055 ceph-mon[80978]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:06.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:05 smithi187 ceph-mon[87698]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:08.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:07 smithi055 ceph-mon[80978]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:07 smithi187 ceph-mon[87698]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:10.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: Reconfiguring crash.smithi187 (monmap changed)... 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:10.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:09 smithi187 ceph-mon[87698]: Reconfiguring daemon crash.smithi187 on smithi187 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: Reconfiguring crash.smithi187 (monmap changed)... 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi187", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:10.092 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:09 smithi055 ceph-mon[80978]: Reconfiguring daemon crash.smithi187 on smithi187 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:08:10.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:11.181 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi187.pmhvpw", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-19T19:08:11.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: Deploying daemon osd.0 on smithi055 2023-12-19T19:08:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: Reconfiguring mgr.smithi187.pmhvpw (monmap changed)... 2023-12-19T19:08:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: Reconfiguring daemon mgr.smithi187.pmhvpw on smithi187 2023-12-19T19:08:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:08:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T19:08:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:12.087 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: Deploying daemon osd.0 on smithi055 2023-12-19T19:08:12.087 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: Reconfiguring mgr.smithi187.pmhvpw (monmap changed)... 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: Reconfiguring daemon mgr.smithi187.pmhvpw on smithi187 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-19T19:08:12.088 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:12.961 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:12 smithi187 ceph-mon[87698]: Reconfiguring mon.smithi187 (monmap changed)... 2023-12-19T19:08:12.961 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:12 smithi187 ceph-mon[87698]: Reconfiguring daemon mon.smithi187 on smithi187 2023-12-19T19:08:12.968 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:12 smithi055 ceph-mon[80978]: Reconfiguring mon.smithi187 (monmap changed)... 2023-12-19T19:08:12.968 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:12 smithi055 ceph-mon[80978]: Reconfiguring daemon mon.smithi187 on smithi187 2023-12-19T19:08:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:13 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:14.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:13 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:15.228 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T19:08:15.228 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T19:08:15.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T19:08:15.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T19:08:15.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T19:08:15.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T19:08:15.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:14 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi055.front.sepia.ceph.com:9093"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi055.front.sepia.ceph.com:3000"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi055.front.sepia.ceph.com:9095"}]: dispatch 2023-12-19T19:08:15.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:14 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:15 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.136 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:15 smithi055 ceph-mon[80978]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:16.137 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:15 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.137 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:15 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:15 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:15 smithi187 ceph-mon[87698]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:15 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:15 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:18.155 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:17 smithi055 ceph-mon[80978]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:18.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:17 smithi187 ceph-mon[87698]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:19.275 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.272+0000 7f1116ffd700 1 -- 172.21.15.55:0/3047700360 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f112002b150 con 0x7f110c068ed0 2023-12-19T19:08:19.275 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 0 on host 'smithi055' 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 msgr2=0x7f110c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 0x7f110c06b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1110000c00 tx=0x7f1110009e30 comp rx=0 tx=0).stop 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 msgr2=0x7f11200fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f111c002a50 tx=0x7f111c0046b0 comp rx=0 tx=0).stop 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 shutdown_connections 2023-12-19T19:08:19.277 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f110c068ed0 0x7f110c06b380 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:19.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11200fd740 0x7f11200fd1d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:19.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 --2- 172.21.15.55:0/3047700360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1120100060 0x7f11200fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:19.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 >> 172.21.15.55:0/3047700360 conn(0x7f11200f71c0 msgr2=0x7f11200f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:08:19.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.274+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 shutdown_connections 2023-12-19T19:08:19.278 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:19.275+0000 7f112838a700 1 -- 172.21.15.55:0/3047700360 wait complete. 2023-12-19T19:08:20.084 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:19 smithi055 ceph-mon[80978]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:20.084 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:20.084 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:20.084 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:19 smithi055 ceph-mon[80978]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-19T19:08:20.113 DEBUG:teuthology.orchestra.run.smithi055:osd.0> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.0.service 2023-12-19T19:08:20.115 INFO:tasks.cephadm:Deploying osd.1 on smithi055 with /dev/vg_nvme/lv_3... 2023-12-19T19:08:20.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-19T19:08:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:19 smithi187 ceph-mon[87698]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:20.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:19 smithi187 ceph-mon[87698]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-19T19:08:20.528 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:08:20.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:20.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:20 smithi055 ceph-mon[80978]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-19T19:08:20.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:20 smithi055 ceph-mon[80978]: osdmap e7: 1 total, 0 up, 1 in 2023-12-19T19:08:20.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:20.935 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:20 smithi055 ceph-mon[80978]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:08:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:20 smithi187 ceph-mon[87698]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-19T19:08:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:20 smithi187 ceph-mon[87698]: osdmap e7: 1 total, 0 up, 1 in 2023-12-19T19:08:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:20 smithi187 ceph-mon[87698]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:08:21.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:08:21.252+0000 7f886127e700 -1 osd.0 0 waiting for initial osdmap 2023-12-19T19:08:21.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:08:21.260+0000 7f8859874700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:08:22.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-mon[80978]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:22.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-mon[80978]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:08:22.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-mon[80978]: osdmap e8: 1 total, 0 up, 1 in 2023-12-19T19:08:22.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:22.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:21 smithi187 ceph-mon[87698]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:21 smithi187 ceph-mon[87698]: from='osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:08:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:21 smithi187 ceph-mon[87698]: osdmap e8: 1 total, 0 up, 1 in 2023-12-19T19:08:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:22.923 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-19T19:08:22.923 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-19T19:08:22.923 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2023-12-19T19:08:22.924 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2023-12-19T19:08:22.924 INFO:teuthology.orchestra.run.smithi055.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108831 s, 963 MB/s 2023-12-19T19:08:22.924 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 2023-12-19T19:08:22.924 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2023-12-19T19:08:23.511 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:08:23.511 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:08:23.511 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] boot 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: osdmap e9: 1 total, 1 up, 1 in 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:23.512 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:23 smithi055 ceph-mon[80978]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:23.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_3 2023-12-19T19:08:23.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:08:23.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: osd.0 [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] boot 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: osdmap e9: 1 total, 1 up, 1 in 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-19T19:08:23.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:23 smithi187 ceph-mon[87698]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-19T19:08:24.500 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:08:24.717 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:24 smithi055 ceph-mon[80978]: osdmap e10: 1 total, 1 up, 1 in 2023-12-19T19:08:24.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:24 smithi187 ceph-mon[87698]: osdmap e10: 1 total, 1 up, 1 in 2023-12-19T19:08:25.721 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:25 smithi055 ceph-mon[80978]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:25.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:25 smithi187 ceph-mon[87698]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:27.042 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 -- 172.21.15.55:0/83208798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 msgr2=0x7f148c102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 --2- 172.21.15.55:0/83208798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c102430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1480004940 tx=0x7f1480009e30 comp rx=0 tx=0).stop 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 -- 172.21.15.55:0/83208798 shutdown_connections 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 --2- 172.21.15.55:0/83208798 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c102430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 --2- 172.21.15.55:0/83208798 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 -- 172.21.15.55:0/83208798 >> 172.21.15.55:0/83208798 conn(0x7f148c0f71d0 msgr2=0x7f148c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:08:27.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 -- 172.21.15.55:0/83208798 shutdown_connections 2023-12-19T19:08:27.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.040+0000 7f1491925700 1 -- 172.21.15.55:0/83208798 wait complete. 2023-12-19T19:08:27.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.041+0000 7f1491925700 1 Processor -- start 2023-12-19T19:08:27.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.041+0000 7f1491925700 1 -- start start 2023-12-19T19:08:27.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.041+0000 7f1491925700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:27.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f1491925700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f1491925700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c10dfd0 con 0x7f148c100050 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f1491925700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148c10e870 con 0x7f148c0fd730 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.55:59112/0 (socket says 172.21.15.55:59112) 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c10d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:58940/0 (socket says 172.21.15.55:58940) 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 -- 172.21.15.55:0/190308936 learned_addr learned my addr 172.21.15.55:0/190308936 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 -- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 msgr2=0x7f148c10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 -- 172.21.15.55:0/190308936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1480005040 con 0x7f148c0fd730 2023-12-19T19:08:27.045 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f148affd700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f147c00eb40 tx=0x7f147c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147c007cc0 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f147c00f040 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f147c00c990 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.042+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148c10eaf0 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.043+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148c10efe0 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.043+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148c04bb80 con 0x7f148c0fd730 2023-12-19T19:08:27.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.044+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f147c00cc00 con 0x7f148c0fd730 2023-12-19T19:08:27.047 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.045+0000 7f14737fe700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 0x7f147406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:27.047 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.045+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f147c085b70 con 0x7f148c0fd730 2023-12-19T19:08:27.047 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.045+0000 7f148a7fc700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 0x7f147406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:27.047 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.045+0000 7f148a7fc700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 0x7f147406b380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14800135e0 tx=0x7f1480009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:08:27.051 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.049+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f147c04da00 con 0x7f148c0fd730 2023-12-19T19:08:27.267 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:27.265+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f148c10f710 con 0x7f1474068ed0 2023-12-19T19:08:27.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi055 2023-12-19T19:08:27.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:27.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:27.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:27.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi055 to 12869M 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:08:27.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi055 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi055 to 12869M 2023-12-19T19:08:28.079 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:08:28.080 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:29.014 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:28 smithi055 ceph-mon[80978]: from='client.24111 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:08:29.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:28 smithi187 ceph-mon[87698]: from='client.24111 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:08:29.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:29.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:29.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:29 smithi055 ceph-mon[80978]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:30.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:30.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:30.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:29 smithi187 ceph-mon[87698]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:31.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:30 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2895311854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34d9690-2f84-4b5d-9aab-47539ed51db7"}]: dispatch 2023-12-19T19:08:31.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:30 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2895311854' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c34d9690-2f84-4b5d-9aab-47539ed51db7"}]': finished 2023-12-19T19:08:31.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:30 smithi055 ceph-mon[80978]: osdmap e11: 2 total, 1 up, 2 in 2023-12-19T19:08:31.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:30 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:31.116 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:30 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3239266077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:08:31.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:30 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2895311854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c34d9690-2f84-4b5d-9aab-47539ed51db7"}]: dispatch 2023-12-19T19:08:31.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:30 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2895311854' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c34d9690-2f84-4b5d-9aab-47539ed51db7"}]': finished 2023-12-19T19:08:31.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:30 smithi187 ceph-mon[87698]: osdmap e11: 2 total, 1 up, 2 in 2023-12-19T19:08:31.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:30 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:31.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:30 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3239266077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:08:32.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:31 smithi055 ceph-mon[80978]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:32.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:31 smithi187 ceph-mon[87698]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:33 smithi055 ceph-mon[80978]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:33.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:33 smithi187 ceph-mon[87698]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:35.201 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:35.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:36.213 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:35 smithi055 ceph-mon[80978]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:36.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:35 smithi187 ceph-mon[87698]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:38.032 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:37 smithi055 ceph-mon[80978]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:38.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:37 smithi187 ceph-mon[87698]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:39.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:38 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-19T19:08:39.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:38 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:39.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:38 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-19T19:08:39.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:38 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:40.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:39 smithi055 ceph-mon[80978]: Deploying daemon osd.1 on smithi055 2023-12-19T19:08:40.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:39 smithi055 ceph-mon[80978]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:40.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:39 smithi187 ceph-mon[87698]: Deploying daemon osd.1 on smithi055 2023-12-19T19:08:40.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:39 smithi187 ceph-mon[87698]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:42.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:41 smithi055 ceph-mon[80978]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:42.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:41 smithi187 ceph-mon[87698]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:44.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:44.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:43 smithi055 ceph-mon[80978]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:44.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:44.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:43 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:44.142 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:44.143 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:43 smithi187 ceph-mon[87698]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:44.143 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:44.143 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:43 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:46.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:45 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:46.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:45 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:46.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:45 smithi055 ceph-mon[80978]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:46.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:45 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:46.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:45 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:46.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:45 smithi187 ceph-mon[87698]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:46.176 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.174+0000 7f14737fe700 1 -- 172.21.15.55:0/190308936 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f148c10f710 con 0x7f1474068ed0 2023-12-19T19:08:46.177 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 1 on host 'smithi055' 2023-12-19T19:08:46.178 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 msgr2=0x7f147406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 0x7f147406b380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f14800135e0 tx=0x7f1480009e30 comp rx=0 tx=0).stop 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 msgr2=0x7f148c10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f147c00eb40 tx=0x7f147c00b040 comp rx=0 tx=0).stop 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 shutdown_connections 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1474068ed0 0x7f147406b380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f148c0fd730 0x7f148c10d490 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 --2- 172.21.15.55:0/190308936 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f148c100050 0x7f148c10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 >> 172.21.15.55:0/190308936 conn(0x7f148c0f71d0 msgr2=0x7f148c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 shutdown_connections 2023-12-19T19:08:46.179 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:46.176+0000 7f1491925700 1 -- 172.21.15.55:0/190308936 wait complete. 2023-12-19T19:08:46.776 DEBUG:teuthology.orchestra.run.smithi055:osd.1> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.1.service 2023-12-19T19:08:46.778 INFO:tasks.cephadm:Deploying osd.2 on smithi055 with /dev/vg_nvme/lv_2... 2023-12-19T19:08:46.778 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-19T19:08:47.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:47 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:47.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:47 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:47.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:47 smithi055 ceph-mon[80978]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-19T19:08:47.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:47 smithi055 ceph-mon[80978]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:47.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:47 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:47.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:47 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:47.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:47 smithi187 ceph-mon[87698]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-19T19:08:47.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:47 smithi187 ceph-mon[87698]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:48.180 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:08:48.456 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-mon[80978]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-19T19:08:48.456 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-mon[80978]: osdmap e12: 2 total, 1 up, 2 in 2023-12-19T19:08:48.456 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:48.456 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-mon[80978]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:08:48.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:08:48.139+0000 7fa133803700 -1 osd.1 0 waiting for initial osdmap 2023-12-19T19:08:48.457 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:08:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:08:48.146+0000 7fa12c5fa700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:08:48.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:48 smithi187 ceph-mon[87698]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-19T19:08:48.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:48 smithi187 ceph-mon[87698]: osdmap e12: 2 total, 1 up, 2 in 2023-12-19T19:08:48.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:48 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:48.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:48 smithi187 ceph-mon[87698]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:08:49.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:49 smithi055 ceph-mon[80978]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:08:49.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:49 smithi055 ceph-mon[80978]: osdmap e13: 2 total, 1 up, 2 in 2023-12-19T19:08:49.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:49.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:49.348 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:49 smithi055 ceph-mon[80978]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:49.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:49 smithi187 ceph-mon[87698]: from='osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:08:49.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:49 smithi187 ceph-mon[87698]: osdmap e13: 2 total, 1 up, 2 in 2023-12-19T19:08:49.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:49.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:49.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:49 smithi187 ceph-mon[87698]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10+0 records in 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr:10+0 records out 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129145 s, 812 MB/s 2023-12-19T19:08:50.174 INFO:teuthology.orchestra.run.smithi055.stderr:--> Zapping successful for: 2023-12-19T19:08:50.197 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] boot 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: osdmap e14: 2 total, 2 up, 2 in 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:50.198 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:50 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: osd.1 [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] boot 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: osdmap e14: 2 total, 2 up, 2 in 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-19T19:08:50.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:50 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:08:50.764 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi055:vg_nvme/lv_2 2023-12-19T19:08:51.475 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:08:51.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:51 smithi055 ceph-mon[80978]: osdmap e15: 2 total, 2 up, 2 in 2023-12-19T19:08:51.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:51 smithi055 ceph-mon[80978]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:51.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:51 smithi187 ceph-mon[87698]: osdmap e15: 2 total, 2 up, 2 in 2023-12-19T19:08:51.562 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:51 smithi187 ceph-mon[87698]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:52.749 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:52.749 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:52.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:52.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:53.159 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 -- 172.21.15.55:0/493219723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 msgr2=0x7fe9680fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:53.159 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/493219723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe9680fd150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe950004940 tx=0x7fe950009e30 comp rx=0 tx=0).stop 2023-12-19T19:08:53.159 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 -- 172.21.15.55:0/493219723 shutdown_connections 2023-12-19T19:08:53.159 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/493219723 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe9680fd150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:53.159 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/493219723 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 0x7fe9680fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 -- 172.21.15.55:0/493219723 >> 172.21.15.55:0/493219723 conn(0x7fe9680f7190 msgr2=0x7fe9680f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 -- 172.21.15.55:0/493219723 shutdown_connections 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.155+0000 7fe96d0b5700 1 -- 172.21.15.55:0/493219723 wait complete. 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 Processor -- start 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 -- start start 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 0x7fe96810f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968110210 con 0x7fe9680fcd00 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96d0b5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe968110ab0 con 0x7fe9680fbaf0 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41208/0 (socket says 172.21.15.55:41208) 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 -- 172.21.15.55:0/2002170807 learned_addr learned my addr 172.21.15.55:0/2002170807 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:08:53.160 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe966d9d700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 0x7fe96810f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 -- 172.21.15.55:0/2002170807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 msgr2=0x7fe96810f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 0x7fe96810f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.156+0000 7fe96659c700 1 -- 172.21.15.55:0/2002170807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950005040 con 0x7fe9680fcd00 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe96659c700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe950000c00 tx=0x7fe950009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe950021070 con 0x7fe9680fcd00 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe95001c400 con 0x7fe9680fcd00 2023-12-19T19:08:53.161 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe968110d30 con 0x7fe9680fcd00 2023-12-19T19:08:53.162 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe95000e070 con 0x7fe9680fcd00 2023-12-19T19:08:53.162 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.157+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968111210 con 0x7fe9680fcd00 2023-12-19T19:08:53.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.158+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96804bb80 con 0x7fe9680fcd00 2023-12-19T19:08:53.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.158+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fe95001c9c0 con 0x7fe9680fcd00 2023-12-19T19:08:53.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.159+0000 7fe95f7fe700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 0x7fe95406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:08:53.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.159+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fe9500872f0 con 0x7fe9680fcd00 2023-12-19T19:08:53.163 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.159+0000 7fe966d9d700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 0x7fe95406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:08:53.164 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.159+0000 7fe966d9d700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 0x7fe95406b380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9680fcb60 tx=0x7fe95800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:08:53.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.163+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe950057f20 con 0x7fe9680fcd00 2023-12-19T19:08:53.372 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:08:53.370+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe968111660 con 0x7fe954068ed0 2023-12-19T19:08:53.766 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:53 smithi055 ceph-mon[80978]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:53.766 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:53 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:08:53.766 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:53 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:08:53.766 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:53 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:53 smithi187 ceph-mon[87698]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:53 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:08:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:53 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:08:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:53 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:54.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:54 smithi187 ceph-mon[87698]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:08:55.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:54 smithi055 ceph-mon[80978]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi055:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:08:55.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:55 smithi187 ceph-mon[87698]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:56.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:55 smithi055 ceph-mon[80978]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:57.547 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi055 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi055 to 6434M 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:57.548 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:57 smithi055 ceph-mon[80978]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:57.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi055 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi055 to 6434M 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:08:57.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:08:57.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:08:57.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:08:57.842 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:57 smithi187 ceph-mon[87698]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:08:59.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:58 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3294523779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c"}]: dispatch 2023-12-19T19:08:59.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:58 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3294523779' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c"}]': finished 2023-12-19T19:08:59.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:58 smithi055 ceph-mon[80978]: osdmap e16: 3 total, 2 up, 3 in 2023-12-19T19:08:59.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:08:59.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:58 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2625853643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:08:59.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:58 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3294523779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c"}]: dispatch 2023-12-19T19:08:59.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:58 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3294523779' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c"}]': finished 2023-12-19T19:08:59.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:58 smithi187 ceph-mon[87698]: osdmap e16: 3 total, 2 up, 3 in 2023-12-19T19:08:59.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:08:59.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:58 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2625853643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:08:59.877 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:08:59 smithi055 ceph-mon[80978]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:00.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:08:59 smithi187 ceph-mon[87698]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:02.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:01 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:01 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:03.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:03 smithi187 ceph-mon[87698]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:03.916 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:03 smithi055 ceph-mon[80978]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:05.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:05.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:06.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:05 smithi055 ceph-mon[80978]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:06.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:05 smithi187 ceph-mon[87698]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:07.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-19T19:09:07.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:07.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:06 smithi055 ceph-mon[80978]: Deploying daemon osd.2 on smithi055 2023-12-19T19:09:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-19T19:09:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:06 smithi187 ceph-mon[87698]: Deploying daemon osd.2 on smithi055 2023-12-19T19:09:08.117 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:07 smithi055 ceph-mon[80978]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:07 smithi187 ceph-mon[87698]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:10.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:09 smithi055 ceph-mon[80978]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:09 smithi187 ceph-mon[87698]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:12.100 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:11 smithi187 ceph-mon[87698]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:12.100 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:12.100 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:12.100 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:12.152 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:11 smithi055 ceph-mon[80978]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:12.152 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:12.152 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:12.152 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:13.711 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:13 smithi055 ceph-mon[80978]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:13.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:13 smithi187 ceph-mon[87698]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:14.646 INFO:teuthology.orchestra.run.smithi055.stdout:Created osd(s) 2 on host 'smithi055' 2023-12-19T19:09:14.647 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.644+0000 7fe95f7fe700 1 -- 172.21.15.55:0/2002170807 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe968111660 con 0x7fe954068ed0 2023-12-19T19:09:14.648 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 msgr2=0x7fe95406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:14.648 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 0x7fe95406b380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe9680fcb60 tx=0x7fe95800b040 comp rx=0 tx=0).stop 2023-12-19T19:09:14.648 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 msgr2=0x7fe96810fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:14.648 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe950000c00 tx=0x7fe950009e30 comp rx=0 tx=0).stop 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 shutdown_connections 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe954068ed0 0x7fe95406b380 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9680fbaf0 0x7fe96810f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 --2- 172.21.15.55:0/2002170807 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9680fcd00 0x7fe96810fc10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 >> 172.21.15.55:0/2002170807 conn(0x7fe9680f7190 msgr2=0x7fe968065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 shutdown_connections 2023-12-19T19:09:14.649 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:09:14.646+0000 7fe96d0b5700 1 -- 172.21.15.55:0/2002170807 wait complete. 2023-12-19T19:09:15.272 DEBUG:teuthology.orchestra.run.smithi055:osd.2> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.2.service 2023-12-19T19:09:15.275 INFO:tasks.cephadm:Deploying osd.3 on smithi187 with /dev/vg_nvme/lv_4... 2023-12-19T19:09:15.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-19T19:09:15.534 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:09:15.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:09:15.324+0000 7fb1ebba0540 -1 osd.2 0 log_to_monitors true 2023-12-19T19:09:15.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:15 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:15.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:15 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:15.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:15 smithi187 ceph-mon[87698]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:15.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:15 smithi187 ceph-mon[87698]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-19T19:09:15.898 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:15 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:15.898 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:15 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:15.898 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:15 smithi055 ceph-mon[80978]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:15.898 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:15 smithi055 ceph-mon[80978]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-19T19:09:16.918 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:16 smithi187 ceph-mon[87698]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-19T19:09:16.918 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:16 smithi187 ceph-mon[87698]: osdmap e17: 3 total, 2 up, 3 in 2023-12-19T19:09:16.918 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:16 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:16.918 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:16 smithi187 ceph-mon[87698]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:09:17.053 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:09:16.640+0000 7fb1e2385700 -1 osd.2 0 waiting for initial osdmap 2023-12-19T19:09:17.054 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:09:16.652+0000 7fb1db97d700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:09:17.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-mon[80978]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-19T19:09:17.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-mon[80978]: osdmap e17: 3 total, 2 up, 3 in 2023-12-19T19:09:17.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:17.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:16 smithi055 ceph-mon[80978]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]: dispatch 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114842 s, 913 MB/s 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 2023-12-19T19:09:17.411 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-12-19T19:09:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:17 smithi055 ceph-mon[80978]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:09:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:17 smithi055 ceph-mon[80978]: osdmap e18: 3 total, 2 up, 3 in 2023-12-19T19:09:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:17 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:17 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:17.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:17 smithi055 ceph-mon[80978]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:18.045 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:17 smithi187 ceph-mon[87698]: from='osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi055", "root=default"]}]': finished 2023-12-19T19:09:18.045 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:17 smithi187 ceph-mon[87698]: osdmap e18: 3 total, 2 up, 3 in 2023-12-19T19:09:18.045 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:17 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:18.046 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:17 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:18.046 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:17 smithi187 ceph-mon[87698]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-19T19:09:18.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2023-12-19T19:09:18.325 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:09:18.740 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:09:18.740 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:09:18.740 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] boot 2023-12-19T19:09:18.740 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: osdmap e19: 3 total, 3 up, 3 in 2023-12-19T19:09:18.741 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:18.741 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:18.741 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:18 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:19.005 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: osd.2 [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] boot 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: osdmap e19: 3 total, 3 up, 3 in 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:19.006 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:18 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:19.809 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:19 smithi055 ceph-mon[80978]: osdmap e20: 3 total, 3 up, 3 in 2023-12-19T19:09:19.809 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:19 smithi055 ceph-mon[80978]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:19.809 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 -- 172.21.15.187:0/2825524681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 msgr2=0x7f5aa80fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/2825524681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa80fbf00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4004940 tx=0x7f5aa4009e30 comp rx=0 tx=0).stop 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 -- 172.21.15.187:0/2825524681 shutdown_connections 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/2825524681 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa80fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/2825524681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa80fbf00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:19.876 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 -- 172.21.15.187:0/2825524681 >> 172.21.15.187:0/2825524681 conn(0x7f5aa80f7190 msgr2=0x7f5aa80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 -- 172.21.15.187:0/2825524681 shutdown_connections 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.872+0000 7f5ab05b3700 1 -- 172.21.15.187:0/2825524681 wait complete. 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 Processor -- start 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 -- start start 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa810c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa810cff0 con 0x7f5aa80fcd00 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5ab05b3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa810d890 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:54538/0 (socket says 172.21.15.187:54538) 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 -- 172.21.15.187:0/1337761307 learned_addr learned my addr 172.21.15.187:0/1337761307 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aadb4e700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa810c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:19.877 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 -- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 msgr2=0x7f5aa810c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa810c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 -- 172.21.15.187:0/1337761307 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4005040 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aadb4e700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa810c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.873+0000 7f5aae34f700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4004f90 tx=0x7f5aa4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.874+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa400e040 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.874+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5aa401e8d0 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.874+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa810db10 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.874+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5aa4013780 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.878 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.874+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa810dff0 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.875+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa804bb80 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.875+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f5aa401ea40 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.875+0000 7f5a9effd700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 0x7f5a9406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.876+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f5aa408a460 con 0x7f5aa80fbaf0 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.876+0000 7f5aadb4e700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 0x7f5a9406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:19.879 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.876+0000 7f5aadb4e700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 0x7f5a9406b380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009fd0 tx=0x7f5a98009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:09:19.882 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:19.880+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5aa405ace0 con 0x7f5aa80fbaf0 2023-12-19T19:09:20.072 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:20.070+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5aa810e400 con 0x7f5a94068ed0 2023-12-19T19:09:20.072 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:19 smithi187 ceph-mon[87698]: osdmap e20: 3 total, 3 up, 3 in 2023-12-19T19:09:20.072 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:19 smithi187 ceph-mon[87698]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:20.072 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: osdmap e21: 3 total, 3 up, 3 in 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:09:20.758 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: osdmap e21: 3 total, 3 up, 3 in 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:09:20.931 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:21.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119151]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T19:09:21.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119151]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:21.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119151]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:21.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119151]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:21.693 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:21.690+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5aa4057620 con 0x7f5aa80fbaf0 2023-12-19T19:09:21.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:21 smithi187 ceph-mon[87698]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:09:21.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-19T19:09:21.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:21 smithi187 ceph-mon[87698]: osdmap e22: 3 total, 3 up, 3 in 2023-12-19T19:09:21.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:21 smithi187 ceph-mon[87698]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:21.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:21.969 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:21 smithi055 ceph-mon[80978]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:09:21.969 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-19T19:09:21.969 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:21 smithi055 ceph-mon[80978]: osdmap e22: 3 total, 3 up, 3 in 2023-12-19T19:09:21.969 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:21 smithi055 ceph-mon[80978]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:21.969 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:22.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119254]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T19:09:22.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119254]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:22.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119254]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:22.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:09:21 smithi055 sudo[119254]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:22.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119343]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T19:09:22.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:22.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119343]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:22.803 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119343]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi055 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi055 to 4289M 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: osdmap e23: 3 total, 3 up, 3 in 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: mgrmap e19: smithi055.dwclvp(active, since 2m), standbys: smithi187.pmhvpw 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:22.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/3644033853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]: dispatch 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]: dispatch 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]': finished 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: osdmap e24: 4 total, 3 up, 4 in 2023-12-19T19:09:22.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:22 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi055 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi055 to 4289M 2023-12-19T19:09:23.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: osdmap e23: 3 total, 3 up, 3 in 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: mgrmap e19: smithi055.dwclvp(active, since 2m), standbys: smithi187.pmhvpw 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/3644033853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]: dispatch 2023-12-19T19:09:23.092 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]: dispatch 2023-12-19T19:09:23.093 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0ea8c34-b465-466e-a827-d096715d37f2"}]': finished 2023-12-19T19:09:23.093 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: osdmap e24: 4 total, 3 up, 4 in 2023-12-19T19:09:23.093 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:22 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:23.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119370]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T19:09:23.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119370]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:23.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119370]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:23.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:09:22 smithi055 sudo[119370]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:23.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119469]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T19:09:23.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119469]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:23.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119469]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:23.556 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119469]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:24.002 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:23 smithi055 ceph-mon[80978]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:24.002 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:23 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2734100725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:09:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119519]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T19:09:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119519]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119519]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:24.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:09:23 smithi055 sudo[119519]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:24.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:23 smithi187 ceph-mon[87698]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:24.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:23 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2734100725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:09:24.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119552]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T19:09:24.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:24.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119552]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:24.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119552]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:25.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119602]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T19:09:25.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:25.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119602]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:25.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:24 smithi055 sudo[119602]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:25.240 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:24 smithi187 sudo[92003]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-19T19:09:25.240 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:24 smithi187 sudo[92003]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:25.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:24 smithi187 sudo[92003]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:25.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 sudo[92003]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:25.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T19:09:25.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T19:09:25.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:09:25.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:09:25.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:09:25.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:09:25.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 sudo[92007]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-19T19:09:25.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 sudo[92007]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-19T19:09:25.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 sudo[92007]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 sudo[92007]: pam_unix(sudo:session): session closed for user root 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi055"}]: dispatch 2023-12-19T19:09:25.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mon metadata", "id": "smithi187"}]: dispatch 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:26.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:26 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:26.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-19T19:09:26.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: pgmap v82: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:26.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-19T19:09:26.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:26.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:26.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:26 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:27.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:27.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:27.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:27 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:27.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:27 smithi187 ceph-mon[87698]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:27.515 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:27.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:27.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:27 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:27.516 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:27 smithi055 ceph-mon[80978]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:30.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:29 smithi187 ceph-mon[87698]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:30.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-19T19:09:30.186 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:29 smithi055 ceph-mon[80978]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-19T19:09:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:31.224 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:30 smithi187 ceph-mon[87698]: Deploying daemon osd.3 on smithi187 2023-12-19T19:09:31.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:30 smithi055 ceph-mon[80978]: Deploying daemon osd.3 on smithi187 2023-12-19T19:09:32.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:31 smithi187 ceph-mon[87698]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-12-19T19:09:32.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:31 smithi055 ceph-mon[80978]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-12-19T19:09:33.767 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:33 smithi187 ceph-mon[87698]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-12-19T19:09:34.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:33 smithi055 ceph-mon[80978]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-12-19T19:09:35.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:35.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:35.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:35.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:35 smithi187 ceph-mon[87698]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-12-19T19:09:35.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:36.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:36.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:36.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:36.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:35 smithi055 ceph-mon[80978]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-12-19T19:09:36.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:38.076 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:37 smithi055 ceph-mon[80978]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-19T19:09:38.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:37 smithi187 ceph-mon[87698]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-19T19:09:38.384 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.381+0000 7f5a9effd700 1 -- 172.21.15.187:0/1337761307 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5aa810e400 con 0x7f5a94068ed0 2023-12-19T19:09:38.384 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 3 on host 'smithi187' 2023-12-19T19:09:38.386 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 msgr2=0x7f5a9406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 0x7f5a9406b380 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5a98009fd0 tx=0x7f5a98009ee0 comp rx=0 tx=0).stop 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 msgr2=0x7f5aa810c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4004f90 tx=0x7f5aa4009e30 comp rx=0 tx=0).stop 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 shutdown_connections 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5a94068ed0 0x7f5a9406b380 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:38.387 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5aa80fbaf0 0x7f5aa810c4b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:38.388 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 --2- 172.21.15.187:0/1337761307 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f5aa80fcd00 0x7f5aa810c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:38.388 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 >> 172.21.15.187:0/1337761307 conn(0x7f5aa80f7190 msgr2=0x7f5aa80fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:09:38.388 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 shutdown_connections 2023-12-19T19:09:38.388 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:38.384+0000 7f5ab05b3700 1 -- 172.21.15.187:0/1337761307 wait complete. 2023-12-19T19:09:39.054 DEBUG:teuthology.orchestra.run.smithi187:osd.3> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.3.service 2023-12-19T19:09:39.057 INFO:tasks.cephadm:Deploying osd.4 on smithi187 with /dev/vg_nvme/lv_3... 2023-12-19T19:09:39.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-19T19:09:39.351 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-12-19T19:09:39.624 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.625 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:39 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:39.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:39 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:40.571 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-mon[87698]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-19T19:09:40.571 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-mon[87698]: osdmap e25: 4 total, 3 up, 4 in 2023-12-19T19:09:40.571 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:40.571 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-mon[87698]: from='osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:09:40.571 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-mon[87698]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:09:40.572 INFO:journalctl@ceph.osd.3.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-3[93286]: 2023-12-19T19:09:40.380+0000 7f06e9663700 -1 osd.3 0 waiting for initial osdmap 2023-12-19T19:09:40.572 INFO:journalctl@ceph.osd.3.smithi187.stdout:Dec 19 19:09:40 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-3[93286]: 2023-12-19T19:09:40.387+0000 7f06e445e700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:09:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:40 smithi055 ceph-mon[80978]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-19T19:09:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:40 smithi055 ceph-mon[80978]: osdmap e25: 4 total, 3 up, 4 in 2023-12-19T19:09:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:40 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:40 smithi055 ceph-mon[80978]: from='osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:09:40.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:40 smithi055 ceph-mon[80978]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:09:41.187 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-19T19:09:41.187 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-19T19:09:41.187 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-12-19T19:09:41.188 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-12-19T19:09:41.188 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145563 s, 720 MB/s 2023-12-19T19:09:41.188 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-12-19T19:09:41.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:41 smithi187 ceph-mon[87698]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:09:41.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:41 smithi187 ceph-mon[87698]: osdmap e26: 4 total, 3 up, 4 in 2023-12-19T19:09:41.424 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:41.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:41 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:41.425 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:41 smithi187 ceph-mon[87698]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:41 smithi055 ceph-mon[80978]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:09:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:41 smithi055 ceph-mon[80978]: osdmap e26: 4 total, 3 up, 4 in 2023-12-19T19:09:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:41 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:41.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:41 smithi055 ceph-mon[80978]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-19T19:09:42.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:09:42.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:09:42.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:42.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] boot 2023-12-19T19:09:42.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: osdmap e27: 4 total, 4 up, 4 in 2023-12-19T19:09:42.510 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:42 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:42.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:09:42.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:09:42.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:42.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: osd.3 [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] boot 2023-12-19T19:09:42.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: osdmap e27: 4 total, 4 up, 4 in 2023-12-19T19:09:42.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:42 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-19T19:09:42.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2023-12-19T19:09:43.732 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:43 smithi187 ceph-mon[87698]: osdmap e28: 4 total, 4 up, 4 in 2023-12-19T19:09:43.733 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:43 smithi187 ceph-mon[87698]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:43.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:43 smithi055 ceph-mon[80978]: osdmap e28: 4 total, 4 up, 4 in 2023-12-19T19:09:43.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:43 smithi055 ceph-mon[80978]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:43.809 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: osdmap e29: 4 total, 4 up, 4 in 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi187 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:44.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi187 to 15936M 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:44.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:44 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: osdmap e29: 4 total, 4 up, 4 in 2023-12-19T19:09:44.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi187 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi187 to 15936M 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:09:44.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:44 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:09:45.621 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 -- 172.21.15.187:0/3260446106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 msgr2=0x7f32580fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 --2- 172.21.15.187:0/3260446106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f32580fd000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3248004940 tx=0x7f3248009e30 comp rx=0 tx=0).stop 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 -- 172.21.15.187:0/3260446106 shutdown_connections 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 --2- 172.21.15.187:0/3260446106 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 0x7f32580fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 --2- 172.21.15.187:0/3260446106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f32580fd000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.617+0000 7f325fa59700 1 -- 172.21.15.187:0/3260446106 >> 172.21.15.187:0/3260446106 conn(0x7f32580f82d0 msgr2=0x7f32580fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 -- 172.21.15.187:0/3260446106 shutdown_connections 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 -- 172.21.15.187:0/3260446106 wait complete. 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 Processor -- start 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 -- start start 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 0x7f325810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:45.622 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325810e010 con 0x7f32580fde00 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325fa59700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325810e8b0 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325d7f5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325d7f5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:49004/0 (socket says 172.21.15.187:49004) 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.618+0000 7f325d7f5700 1 -- 172.21.15.187:0/4058828976 learned_addr learned my addr 172.21.15.187:0/4058828976 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325cff4700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 0x7f325810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325d7f5700 1 -- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 msgr2=0x7f325810da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325d7f5700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 0x7f325810da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325d7f5700 1 -- 172.21.15.187:0/4058828976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248005040 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325d7f5700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3248000c00 tx=0x7f3248009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324800e040 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3248009af0 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325810eb30 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324801c630 con 0x7f32580fcbf0 2023-12-19T19:09:45.623 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.619+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325810f070 con 0x7f32580fcbf0 2023-12-19T19:09:45.624 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.620+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325804bb80 con 0x7f32580fcbf0 2023-12-19T19:09:45.624 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.621+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f324800bc50 con 0x7f32580fcbf0 2023-12-19T19:09:45.624 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.621+0000 7f324dffb700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 0x7f324406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:09:45.624 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.621+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f3248087930 con 0x7f32580fcbf0 2023-12-19T19:09:45.624 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.621+0000 7f325cff4700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 0x7f324406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:09:45.625 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.622+0000 7f325cff4700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 0x7f324406b410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3254009fd0 tx=0x7f325400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:09:45.629 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.626+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3248057fd0 con 0x7f32580fcbf0 2023-12-19T19:09:45.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:45 smithi055 ceph-mon[80978]: osdmap e30: 4 total, 4 up, 4 in 2023-12-19T19:09:45.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:45 smithi055 ceph-mon[80978]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:45.818 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:09:45.815+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f325810f4d0 con 0x7f3244068f60 2023-12-19T19:09:45.818 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:45 smithi187 ceph-mon[87698]: osdmap e30: 4 total, 4 up, 4 in 2023-12-19T19:09:45.818 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:45 smithi187 ceph-mon[87698]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:46.495 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:46 smithi187 ceph-mon[87698]: osdmap e31: 4 total, 4 up, 4 in 2023-12-19T19:09:46.495 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:46 smithi187 ceph-mon[87698]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:09:46.495 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:46 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:09:46.495 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:46 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:09:46.495 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:46 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:46.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:46 smithi055 ceph-mon[80978]: osdmap e31: 4 total, 4 up, 4 in 2023-12-19T19:09:46.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:46 smithi055 ceph-mon[80978]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:09:46.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:46 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:09:46.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:46 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:09:46.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:46 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:47.509 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:47 smithi187 ceph-mon[87698]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-12-19T19:09:47.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:47 smithi055 ceph-mon[80978]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-12-19T19:09:48.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:48 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2413060865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]: dispatch 2023-12-19T19:09:48.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:48 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]: dispatch 2023-12-19T19:09:48.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:48 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]': finished 2023-12-19T19:09:48.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:48 smithi055 ceph-mon[80978]: osdmap e32: 5 total, 4 up, 5 in 2023-12-19T19:09:48.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:48 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:09:48.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:48 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2413060865' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]: dispatch 2023-12-19T19:09:48.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:48 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]: dispatch 2023-12-19T19:09:48.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:48 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6d80173-9176-4bde-a702-f4a43309d5f8"}]': finished 2023-12-19T19:09:48.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:48 smithi187 ceph-mon[87698]: osdmap e32: 5 total, 4 up, 5 in 2023-12-19T19:09:48.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:48 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:09:50.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:49 smithi055 ceph-mon[80978]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-12-19T19:09:50.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:49 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/1612354857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:09:50.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:49 smithi187 ceph-mon[87698]: pgmap v102: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-12-19T19:09:50.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:49 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/1612354857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:09:51.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:50 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:51.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:50 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:09:51.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:51 smithi187 ceph-mon[87698]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-12-19T19:09:52.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:51 smithi055 ceph-mon[80978]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-12-19T19:09:53.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:53 smithi187 ceph-mon[87698]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-12-19T19:09:54.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:53 smithi055 ceph-mon[80978]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-12-19T19:09:56.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:55 smithi187 ceph-mon[87698]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:56.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:55 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-19T19:09:56.232 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:55 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:56.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:55 smithi055 ceph-mon[80978]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:56.305 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:55 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-19T19:09:56.305 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:55 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:09:57.154 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:56 smithi187 ceph-mon[87698]: Deploying daemon osd.4 on smithi187 2023-12-19T19:09:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:56 smithi055 ceph-mon[80978]: Deploying daemon osd.4 on smithi187 2023-12-19T19:09:58.227 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:57 smithi187 ceph-mon[87698]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:09:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:57 smithi055 ceph-mon[80978]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:00.066 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:09:59 smithi187 ceph-mon[87698]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:00.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:09:59 smithi055 ceph-mon[80978]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:01.266 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:00 smithi187 ceph-mon[87698]: overall HEALTH_OK 2023-12-19T19:10:01.266 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:00 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:01.266 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:00 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:01.266 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:00 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:01.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:00 smithi055 ceph-mon[80978]: overall HEALTH_OK 2023-12-19T19:10:01.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:00 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:01.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:00 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:01.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:00 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:02.229 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:01 smithi187 ceph-mon[87698]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:01 smithi055 ceph-mon[80978]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:03.434 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.429+0000 7f324dffb700 1 -- 172.21.15.187:0/4058828976 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f325810f4d0 con 0x7f3244068f60 2023-12-19T19:10:03.434 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 4 on host 'smithi187' 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 msgr2=0x7f324406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 0x7f324406b410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3254009fd0 tx=0x7f325400b040 comp rx=0 tx=0).stop 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 msgr2=0x7f325810d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3248000c00 tx=0x7f3248009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 shutdown_connections 2023-12-19T19:10:03.435 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3244068f60 0x7f324406b410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:03.436 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32580fcbf0 0x7f325810d4d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:03.436 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 --2- 172.21.15.187:0/4058828976 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32580fde00 0x7f325810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:03.436 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.431+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 >> 172.21.15.187:0/4058828976 conn(0x7f32580f82d0 msgr2=0x7f3258101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:03.436 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.432+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 shutdown_connections 2023-12-19T19:10:03.436 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:03.432+0000 7f325fa59700 1 -- 172.21.15.187:0/4058828976 wait complete. 2023-12-19T19:10:04.047 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:03 smithi187 ceph-mon[87698]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:04.047 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:03 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:04.047 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:03 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:04.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:03 smithi055 ceph-mon[80978]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:04.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:03 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:04.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:03 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:04.768 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:04 smithi187 ceph-mon[87698]: from='osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T19:10:04.768 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:04 smithi187 ceph-mon[87698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T19:10:04.792 DEBUG:teuthology.orchestra.run.smithi187:osd.4> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.4.service 2023-12-19T19:10:04.795 INFO:tasks.cephadm:Deploying osd.5 on smithi187 with /dev/vg_nvme/lv_2... 2023-12-19T19:10:04.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-19T19:10:04.906 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:04 smithi055 ceph-mon[80978]: from='osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T19:10:04.906 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:04 smithi055 ceph-mon[80978]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-19T19:10:05.168 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: osdmap e33: 5 total, 4 up, 5 in 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: from='osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:05.926 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:05.926 INFO:journalctl@ceph.osd.4.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-4[100141]: 2023-12-19T19:10:05.619+0000 7f440a92e700 -1 osd.4 0 waiting for initial osdmap 2023-12-19T19:10:05.926 INFO:journalctl@ceph.osd.4.smithi187.stdout:Dec 19 19:10:05 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-4[100141]: 2023-12-19T19:10:05.626+0000 7f4404f28700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: osdmap e33: 5 total, 4 up, 5 in 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: from='osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-19T19:10:06.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: osdmap e34: 5 total, 4 up, 5 in 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:06.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: osdmap e34: 5 total, 4 up, 5 in 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:06.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:07.076 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-19T19:10:07.076 INFO:teuthology.orchestra.run.smithi187.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-19T19:10:07.077 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10+0 records in 2023-12-19T19:10:07.077 INFO:teuthology.orchestra.run.smithi187.stderr:10+0 records out 2023-12-19T19:10:07.077 INFO:teuthology.orchestra.run.smithi187.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141151 s, 743 MB/s 2023-12-19T19:10:07.077 INFO:teuthology.orchestra.run.smithi187.stderr:--> Zapping successful for: 2023-12-19T19:10:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:10:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:10:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] boot 2023-12-19T19:10:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: osdmap e35: 5 total, 5 up, 5 in 2023-12-19T19:10:07.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:07.804 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:07 smithi055 ceph-mon[80978]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:07.837 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:10:07.837 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:10:07.837 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: osd.4 [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] boot 2023-12-19T19:10:07.837 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: osdmap e35: 5 total, 5 up, 5 in 2023-12-19T19:10:07.837 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-19T19:10:07.838 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:07 smithi187 ceph-mon[87698]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:08.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2023-12-19T19:10:08.713 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:08 smithi187 ceph-mon[87698]: osdmap e36: 5 total, 5 up, 5 in 2023-12-19T19:10:08.942 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:10:09.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:08 smithi055 ceph-mon[80978]: osdmap e36: 5 total, 5 up, 5 in 2023-12-19T19:10:09.975 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:09 smithi187 ceph-mon[87698]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:10.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:09 smithi055 ceph-mon[80978]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:11.209 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 -- 172.21.15.187:0/3864825494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fcbf0 msgr2=0x7f32a40fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:11.210 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 --2- 172.21.15.187:0/3864825494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fcbf0 0x7f32a40fd000 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32a0004940 tx=0x7f32a0009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:11.210 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 -- 172.21.15.187:0/3864825494 shutdown_connections 2023-12-19T19:10:11.210 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 --2- 172.21.15.187:0/3864825494 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fde00 0x7f32a40fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:11.210 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 --2- 172.21.15.187:0/3864825494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fcbf0 0x7f32a40fd000 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:11.210 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.207+0000 7f32ab90d700 1 -- 172.21.15.187:0/3864825494 >> 172.21.15.187:0/3864825494 conn(0x7f32a40f82d0 msgr2=0x7f32a40fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 -- 172.21.15.187:0/3864825494 shutdown_connections 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 -- 172.21.15.187:0/3864825494 wait complete. 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 Processor -- start 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 -- start start 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 0x7f32a410d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a410e190 con 0x7f32a40fcbf0 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.208+0000 7f32ab90d700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a410e9d0 con 0x7f32a40fde00 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:53114/0 (socket says 172.21.15.187:53114) 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 -- 172.21.15.187:0/2764603082 learned_addr learned my addr 172.21.15.187:0/2764603082 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 -- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 msgr2=0x7f32a410d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a96a9700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 0x7f32a410d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:11.212 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 0x7f32a410d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 -- 172.21.15.187:0/2764603082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a0005040 con 0x7f32a40fde00 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a96a9700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 0x7f32a410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32a8ea8700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f329400b790 tx=0x7f3294009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329400f770 con 0x7f32a40fde00 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a410ecb0 con 0x7f32a40fde00 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f329400fd80 con 0x7f32a40fde00 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f329400d310 con 0x7f32a40fde00 2023-12-19T19:10:11.213 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.209+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a410f1f0 con 0x7f32a40fde00 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.210+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a404bb80 con 0x7f32a40fde00 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.211+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3294010420 con 0x7f32a40fde00 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.211+0000 7f3299ffb700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 0x7f329006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.211+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f3294086890 con 0x7f32a40fde00 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.212+0000 7f32a96a9700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 0x7f329006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:11.214 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.212+0000 7f32a96a9700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 0x7f329006b410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32a0000c00 tx=0x7f32a0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:11.217 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.216+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3294056ce0 con 0x7f32a40fde00 2023-12-19T19:10:11.415 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:11.413+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f32a410f6c0 con 0x7f3290068f60 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi187 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi187 to 7968M 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:11.884 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:10:11.885 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:11 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi187 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi187 to 7968M 2023-12-19T19:10:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-19T19:10:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:11 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:12.798 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:12 smithi187 ceph-mon[87698]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:10:13.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:12 smithi055 ceph-mon[80978]: from='client.24179 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:10:14.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:13 smithi055 ceph-mon[80978]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:13 smithi187 ceph-mon[87698]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:15.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2762244887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]: dispatch 2023-12-19T19:10:15.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]: dispatch 2023-12-19T19:10:15.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]': finished 2023-12-19T19:10:15.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: osdmap e37: 6 total, 5 up, 6 in 2023-12-19T19:10:15.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:15.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:14 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2987716640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:10:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2762244887' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]: dispatch 2023-12-19T19:10:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]: dispatch 2023-12-19T19:10:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc051ac7-0460-497e-a684-63aea173b125"}]': finished 2023-12-19T19:10:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: osdmap e37: 6 total, 5 up, 6 in 2023-12-19T19:10:15.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:15.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:14 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2987716640' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-19T19:10:16.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:15 smithi187 ceph-mon[87698]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:16.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:15 smithi055 ceph-mon[80978]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:18.047 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:17 smithi187 ceph-mon[87698]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:18.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:17 smithi055 ceph-mon[80978]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:20.261 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:19 smithi187 ceph-mon[87698]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:20.261 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:19 smithi055 ceph-mon[80978]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:21.293 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-19T19:10:21.294 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:20 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:21.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-19T19:10:21.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:20 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:22.297 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:21 smithi187 ceph-mon[87698]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:22.297 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:21 smithi187 ceph-mon[87698]: Deploying daemon osd.5 on smithi187 2023-12-19T19:10:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:21 smithi055 ceph-mon[80978]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:22.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:21 smithi055 ceph-mon[80978]: Deploying daemon osd.5 on smithi187 2023-12-19T19:10:23.999 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:23 smithi187 ceph-mon[87698]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:24.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:23 smithi055 ceph-mon[80978]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:26.184 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:25 smithi187 ceph-mon[87698]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:26.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:26.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:26.185 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:25 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:26.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:25 smithi055 ceph-mon[80978]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:26.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:26.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:26.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:25 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:28.231 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:27 smithi187 ceph-mon[87698]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:28.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:27 smithi055 ceph-mon[80978]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:29.632 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.629+0000 7f3299ffb700 1 -- 172.21.15.187:0/2764603082 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f32a410f6c0 con 0x7f3290068f60 2023-12-19T19:10:29.632 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 5 on host 'smithi187' 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 msgr2=0x7f329006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 0x7f329006b410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f32a0000c00 tx=0x7f32a0009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 msgr2=0x7f32a410db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f329400b790 tx=0x7f3294009250 comp rx=0 tx=0).stop 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 shutdown_connections 2023-12-19T19:10:29.634 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3290068f60 0x7f329006b410 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f32a40fcbf0 0x7f32a410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 --2- 172.21.15.187:0/2764603082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f32a40fde00 0x7f32a410db00 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f329400b790 tx=0x7f3294009250 comp rx=0 tx=0).stop 2023-12-19T19:10:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 >> 172.21.15.187:0/2764603082 conn(0x7f32a40f82d0 msgr2=0x7f32a4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 shutdown_connections 2023-12-19T19:10:29.635 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:29.632+0000 7f32ab90d700 1 -- 172.21.15.187:0/2764603082 wait complete. 2023-12-19T19:10:30.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:29 smithi187 ceph-mon[87698]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:30.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:30.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:29 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:30.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:29 smithi187 ceph-mon[87698]: from='osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T19:10:30.208 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:29 smithi187 ceph-mon[87698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T19:10:30.232 DEBUG:teuthology.orchestra.run.smithi187:osd.5> sudo journalctl -f -n 0 -u ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.5.service 2023-12-19T19:10:30.235 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2023-12-19T19:10:30.235 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-19T19:10:30.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:29 smithi055 ceph-mon[80978]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:30.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:30.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:29 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:30.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:29 smithi055 ceph-mon[80978]: from='osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T19:10:30.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:29 smithi055 ceph-mon[80978]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-19T19:10:30.521 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:31.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:31.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:31.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-19T19:10:31.553 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: osdmap e38: 6 total, 5 up, 6 in 2023-12-19T19:10:31.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:31.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:31.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:31.554 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:31 smithi055 ceph-mon[80978]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: osdmap e38: 6 total, 5 up, 6 in 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:31.561 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]: dispatch 2023-12-19T19:10:31.562 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-mon[87698]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-19T19:10:31.840 INFO:journalctl@ceph.osd.5.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-5[106596]: 2023-12-19T19:10:31.601+0000 7fb34559f700 -1 osd.5 0 waiting for initial osdmap 2023-12-19T19:10:31.840 INFO:journalctl@ceph.osd.5.smithi187.stdout:Dec 19 19:10:31 smithi187 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-5[106596]: 2023-12-19T19:10:31.611+0000 7fb34139c700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-19T19:10:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 -- 172.21.15.55:0/2215573574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fcbf0 msgr2=0x7fa9140fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 --2- 172.21.15.55:0/2215573574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fcbf0 0x7fa9140fd000 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa904009ce0 tx=0x7fa904004d10 comp rx=0 tx=0).stop 2023-12-19T19:10:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 -- 172.21.15.55:0/2215573574 shutdown_connections 2023-12-19T19:10:32.086 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 --2- 172.21.15.55:0/2215573574 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fde00 0x7fa9140fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 --2- 172.21.15.55:0/2215573574 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fcbf0 0x7fa9140fd000 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 -- 172.21.15.55:0/2215573574 >> 172.21.15.55:0/2215573574 conn(0x7fa9140f82d0 msgr2=0x7fa9140fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 -- 172.21.15.55:0/2215573574 shutdown_connections 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.080+0000 7fa918a9b700 1 -- 172.21.15.55:0/2215573574 wait complete. 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 Processor -- start 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 -- start start 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 0x7fa91410f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914110410 con 0x7fa9140fde00 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914110cb0 con 0x7fa9140fcbf0 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50584/0 (socket says 172.21.15.55:50584) 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 -- 172.21.15.55:0/3592778959 learned_addr learned my addr 172.21.15.55:0/3592778959 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa91259c700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 0x7fa91410f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:32.087 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 -- 172.21.15.55:0/3592778959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 msgr2=0x7fa91410f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 0x7fa91410f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 -- 172.21.15.55:0/3592778959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904009940 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa91259c700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 0x7fa91410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa911d9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c8c0 tx=0x7fa9080044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa90800a980 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa90800ce60 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.081+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa914110f90 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.082+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9080113a0 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.082+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9141114d0 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.082+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa914101ea0 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.083+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa908012400 con 0x7fa9140fde00 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.083+0000 7fa902ffd700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 0x7fa8fc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:32.088 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.084+0000 7fa91259c700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 0x7fa8fc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:32.089 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.084+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fa908086a90 con 0x7fa9140fde00 2023-12-19T19:10:32.089 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.084+0000 7fa91259c700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 0x7fa8fc06b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa904003400 tx=0x7fa904004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:32.089 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.088+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa908056e40 con 0x7fa9140fde00 2023-12-19T19:10:32.288 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.286+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa9141117a0 con 0x7fa9140fde00 2023-12-19T19:10:32.288 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.286+0000 7fa902ffd700 1 -- 172.21.15.55:0/3592778959 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa9080569d0 con 0x7fa9140fde00 2023-12-19T19:10:32.289 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:32.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 msgr2=0x7fa8fc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 0x7fa8fc06b410 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa904003400 tx=0x7fa904004d10 comp rx=0 tx=0).stop 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 msgr2=0x7fa91410fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa90800c8c0 tx=0x7fa9080044c0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 shutdown_connections 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa8fc068f60 0x7fa8fc06b410 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9140fcbf0 0x7fa91410f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 --2- 172.21.15.55:0/3592778959 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9140fde00 0x7fa91410fe10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 >> 172.21.15.55:0/3592778959 conn(0x7fa9140f82d0 msgr2=0x7fa9140ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 shutdown_connections 2023-12-19T19:10:32.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:32.288+0000 7fa918a9b700 1 -- 172.21.15.55:0/3592778959 wait complete. 2023-12-19T19:10:32.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:32 smithi187 ceph-mon[87698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:10:32.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:32 smithi187 ceph-mon[87698]: osdmap e39: 6 total, 5 up, 6 in 2023-12-19T19:10:32.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:32 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:32.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:32 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:32.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:32 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3592778959' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T19:10:32.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:32 smithi055 ceph-mon[80978]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2023-12-19T19:10:32.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:32 smithi055 ceph-mon[80978]: osdmap e39: 6 total, 5 up, 6 in 2023-12-19T19:10:32.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:32 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:32.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:32 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:32.874 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:32 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3592778959' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T19:10:32.898 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1703013006,"num_in_osds":6,"osd_in_since":1703013013,"num_remapped_pgs":0} 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: purged_snaps scrub starts 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: purged_snaps scrub ok 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] boot 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: osdmap e40: 6 total, 6 up, 6 in 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:33.723 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:33 smithi187 ceph-mon[87698]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:33.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: purged_snaps scrub starts 2023-12-19T19:10:33.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: purged_snaps scrub ok 2023-12-19T19:10:33.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: osd.5 [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] boot 2023-12-19T19:10:33.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: osdmap e40: 6 total, 6 up, 6 in 2023-12-19T19:10:33.805 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-19T19:10:33.806 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:33 smithi055 ceph-mon[80978]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:33.899 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-19T19:10:34.174 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:35.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: osdmap e41: 6 total, 6 up, 6 in 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: Detected new or changed devices on smithi187 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: Adjusting osd_memory_target on smithi187 to 5312M 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:35.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:35.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:35.055 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: osdmap e41: 6 total, 6 up, 6 in 2023-12-19T19:10:35.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: Detected new or changed devices on smithi187 2023-12-19T19:10:35.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: Adjusting osd_memory_target on smithi187 to 5312M 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:10:35.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:10:35.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 -- 172.21.15.55:0/4076463528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c066920 msgr2=0x7f8d4c0ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 --2- 172.21.15.55:0/4076463528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c066920 0x7f8d4c0ff7b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c004940 tx=0x7f8d3c009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 -- 172.21.15.55:0/4076463528 shutdown_connections 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 --2- 172.21.15.55:0/4076463528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c1021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 --2- 172.21.15.55:0/4076463528 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c066920 0x7f8d4c0ff7b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 -- 172.21.15.55:0/4076463528 >> 172.21.15.55:0/4076463528 conn(0x7f8d4c0f71c0 msgr2=0x7f8d4c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 -- 172.21.15.55:0/4076463528 shutdown_connections 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.837+0000 7f8d516db700 1 -- 172.21.15.55:0/4076463528 wait complete. 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 Processor -- start 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 -- start start 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c066920 0x7f8d4c0fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0fbcb0 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d516db700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4c0fbe20 con 0x7f8d4c066920 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:50614/0 (socket says 172.21.15.55:50614) 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 -- 172.21.15.55:0/384578492 learned_addr learned my addr 172.21.15.55:0/384578492 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4affd700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c066920 0x7f8d4c0fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 -- 172.21.15.55:0/384578492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c066920 msgr2=0x7f8d4c0fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c066920 0x7f8d4c0fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.838+0000 7f8d4a7fc700 1 -- 172.21.15.55:0/384578492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c005040 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d4a7fc700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8d34002a50 tx=0x7f8d34004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d34007cc0 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d3400f040 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d3400d990 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4c0fc100 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.839+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4c115590 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.840+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4c1075f0 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.840+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8d34009710 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.841+0000 7f8d437fe700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 0x7f8d3806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.841+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8d340868a0 con 0x7f8d4c0ffd80 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.841+0000 7f8d4affd700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 0x7f8d3806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.841+0000 7f8d4affd700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 0x7f8d3806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00b3b0 tx=0x7f8d3c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:35.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:35.845+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8d34056ac0 con 0x7f8d4c0ffd80 2023-12-19T19:10:36.043 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.041+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8d4c02b110 con 0x7f8d4c0ffd80 2023-12-19T19:10:36.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:35 smithi055 ceph-mon[80978]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:36.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:36.044 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.041+0000 7f8d437fe700 1 -- 172.21.15.55:0/384578492 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8d34056650 con 0x7f8d4c0ffd80 2023-12-19T19:10:36.044 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 msgr2=0x7f8d3806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 0x7f8d3806b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00b3b0 tx=0x7f8d3c009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 msgr2=0x7f8d4c0fb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8d34002a50 tx=0x7f8d34004600 comp rx=0 tx=0).stop 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 shutdown_connections 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8d38068f60 0x7f8d3806b410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8d4c066920 0x7f8d4c0fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 --2- 172.21.15.55:0/384578492 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8d4c0ffd80 0x7f8d4c0fb6b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 >> 172.21.15.55:0/384578492 conn(0x7f8d4c0f71c0 msgr2=0x7f8d4c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 shutdown_connections 2023-12-19T19:10:36.046 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:36.043+0000 7f8d516db700 1 -- 172.21.15.55:0/384578492 wait complete. 2023-12-19T19:10:36.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:35 smithi187 ceph-mon[87698]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:36.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:36.755 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:36 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/384578492' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T19:10:36.785 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1703013032,"num_in_osds":6,"osd_in_since":1703013013,"num_remapped_pgs":0} 2023-12-19T19:10:36.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T19:10:37.059 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:37.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:36 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/384578492' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-19T19:10:37.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:37 smithi055 ceph-mon[80978]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:38.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:37 smithi187 ceph-mon[87698]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:38.619 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 -- 172.21.15.55:0/1305515797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f108006a6a0 msgr2=0x7f108006aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 --2- 172.21.15.55:0/1305515797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f108006a6a0 0x7f108006aa70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1070004940 tx=0x7f1070009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 -- 172.21.15.55:0/1305515797 shutdown_connections 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 --2- 172.21.15.55:0/1305515797 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1080104c40 0x7f10801050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 --2- 172.21.15.55:0/1305515797 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f108006a6a0 0x7f108006aa70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 -- 172.21.15.55:0/1305515797 >> 172.21.15.55:0/1305515797 conn(0x7f1080068fd0 msgr2=0x7f108006e390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.616+0000 7f1085a49700 1 -- 172.21.15.55:0/1305515797 shutdown_connections 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 -- 172.21.15.55:0/1305515797 wait complete. 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 Processor -- start 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 -- start start 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f108006a6a0 0x7f1080122450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:38.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1080122fe0 con 0x7f1080104c40 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1085a49700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108011c330 con 0x7f108006a6a0 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1077fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1077fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:38984/0 (socket says 172.21.15.55:38984) 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.617+0000 7f1077fff700 1 -- 172.21.15.55:0/3661882057 learned_addr learned my addr 172.21.15.55:0/3661882057 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:38.621 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f107effd700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f108006a6a0 0x7f1080122450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1077fff700 1 -- 172.21.15.55:0/3661882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f108006a6a0 msgr2=0x7f1080122450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1077fff700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f108006a6a0 0x7f1080122450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1077fff700 1 -- 172.21.15.55:0/3661882057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1070005040 con 0x7f1080104c40 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1077fff700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1068002a50 tx=0x7f10680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1068007cc0 con 0x7f1080104c40 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108011c610 con 0x7f1080104c40 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f106800f040 con 0x7f1080104c40 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f106800d9c0 con 0x7f1080104c40 2023-12-19T19:10:38.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.618+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f108011cb50 con 0x7f1080104c40 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.619+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1080109ad0 con 0x7f1080104c40 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.620+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f106800db20 con 0x7f1080104c40 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.620+0000 7f10777fe700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 0x7f106c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.620+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1068086790 con 0x7f1080104c40 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.620+0000 7f107effd700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 0x7f106c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:38.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.621+0000 7f107effd700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 0x7f106c06b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f107000b3b0 tx=0x7f1070009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:38.627 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.625+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1068052f20 con 0x7f1080104c40 2023-12-19T19:10:38.825 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.822+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f108004bb80 con 0x7f1080104c40 2023-12-19T19:10:38.825 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.823+0000 7f10777fe700 1 -- 172.21.15.55:0/3661882057 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11091 (secure 0 0 0) 0x7f1068056540 con 0x7f1080104c40 2023-12-19T19:10:38.826 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:38.826 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":41,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","created":"2023-12-19T19:04:57.836701+0000","modified":"2023-12-19T19:10:33.614598+0000","last_up_change":"2023-12-19T19:10:32.592061+0000","last_in_change":"2023-12-19T19:10:13.790913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T19:09:19.364422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e4ca473b-4320-4248-b6b1-fb3182ca21ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6803","nonce":3885599653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6805","nonce":3885599653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6809","nonce":3885599653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6807","nonce":3885599653}]},"public_addr":"172.21.15.55:6803/3885599653","cluster_addr":"172.21.15.55:6805/3885599653","heartbeat_back_addr":"172.21.15.55:6809/3885599653","heartbeat_front_addr":"172.21.15.55:6807/3885599653","state":["exists","up"]},{"osd":1,"uuid":"c34d9690-2f84-4b5d-9aab-47539ed51db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6811","nonce":2714383701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6813","nonce":2714383701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6817","nonce":2714383701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6815","nonce":2714383701}]},"public_addr":"172.21.15.55:6811/2714383701","cluster_addr":"172.21.15.55:6813/2714383701","heartbeat_back_addr":"172.21.15.55:6817/2714383701","heartbeat_front_addr":"172.21.15.55:6815/2714383701","state":["exists","up"]},{"osd":2,"uuid":"a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6819","nonce":3562432537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6821","nonce":3562432537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6825","nonce":3562432537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6823","nonce":3562432537}]},"public_addr":"172.21.15.55:6819/3562432537","cluster_addr":"172.21.15.55:6821/3562432537","heartbeat_back_addr":"172.21.15.55:6825/3562432537","heartbeat_front_addr":"172.21.15.55:6823/3562432537","state":["exists","up"]},{"osd":3,"uuid":"b0ea8c34-b465-466e-a827-d096715d37f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6801","nonce":1951987134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6803","nonce":1951987134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6807","nonce":1951987134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6805","nonce":1951987134}]},"public_addr":"172.21.15.187:6801/1951987134","cluster_addr":"172.21.15.187:6803/1951987134","heartbeat_back_addr":"172.21.15.187:6807/1951987134","heartbeat_front_addr":"172.21.15.187:6805/1951987134","state":["exists","up"]},{"osd":4,"uuid":"d6d80173-9176-4bde-a702-f4a43309d5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6809","nonce":2860720480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6811","nonce":2860720480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6815","nonce":2860720480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6813","nonce":2860720480}]},"public_addr":"172.21.15.187:6809/2860720480","cluster_addr":"172.21.15.187:6811/2860720480","heartbeat_back_addr":"172.21.15.187:6815/2860720480","heartbeat_front_addr":"172.21.15.187:6813/2860720480","state":["exists","up"]},{"osd":5,"uuid":"dc051ac7-0460-497e-a684-63aea173b125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6817","nonce":1722753525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6819","nonce":1722753525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6823","nonce":1722753525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6821","nonce":1722753525}]},"public_addr":"172.21.15.187:6817/1722753525","cluster_addr":"172.21.15.187:6819/1722753525","heartbeat_back_addr":"172.21.15.187:6823/1722753525","heartbeat_front_addr":"172.21.15.187:6821/1722753525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:20.662263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:47.762312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:16.369465+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:39.397498+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:05.487467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:30.696534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/1287666552":"2023-12-20T19:07:04.834074+0000","172.21.15.55:6800/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2442422185":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/3109624574":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/907073539":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2629565350":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6800/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/937693294":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/2933015314":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/4084383373":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2133728457":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6800/917343204":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6801/917343204":"2023-12-20T19:05:32.358078+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T19:10:38.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 msgr2=0x7f106c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 0x7f106c06b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f107000b3b0 tx=0x7f1070009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 msgr2=0x7f1080122990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1068002a50 tx=0x7f10680046b0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 shutdown_connections 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f106c068f60 0x7f106c06b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f108006a6a0 0x7f1080122450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 --2- 172.21.15.55:0/3661882057 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1080104c40 0x7f1080122990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 >> 172.21.15.55:0/3661882057 conn(0x7f1080068fd0 msgr2=0x7f108010f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 shutdown_connections 2023-12-19T19:10:38.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:38.825+0000 7f1085a49700 1 -- 172.21.15.55:0/3661882057 wait complete. 2023-12-19T19:10:39.131 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:38 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3661882057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:39.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:38 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3661882057' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:39.513 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T19:09:19.364422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-19T19:10:39.513 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-12-19T19:10:39.782 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:40.157 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:39 smithi055 ceph-mon[80978]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:40.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:39 smithi187 ceph-mon[87698]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 -- 172.21.15.55:0/2810451339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 msgr2=0x7fc4d4102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 --2- 172.21.15.55:0/2810451339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4102fd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4004940 tx=0x7fc4c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 -- 172.21.15.55:0/2810451339 shutdown_connections 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 --2- 172.21.15.55:0/2810451339 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d40fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 --2- 172.21.15.55:0/2810451339 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4102fd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 -- 172.21.15.55:0/2810451339 >> 172.21.15.55:0/2810451339 conn(0x7fc4d40f8790 msgr2=0x7fc4d40faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 -- 172.21.15.55:0/2810451339 shutdown_connections 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 -- 172.21.15.55:0/2810451339 wait complete. 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.257+0000 7fc4da307700 1 Processor -- start 2023-12-19T19:10:41.260 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4da307700 1 -- start start 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4da307700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4da307700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d4113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4da307700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4114200 con 0x7fc4d4102c00 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4da307700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d410d4f0 con 0x7fc4d41035a0 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39002/0 (socket says 172.21.15.55:39002) 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 -- 172.21.15.55:0/4262595311 learned_addr learned my addr 172.21.15.55:0/4262595311 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d37fe700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d4113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 -- 172.21.15.55:0/4262595311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 msgr2=0x7fc4d4113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d4113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.262 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 -- 172.21.15.55:0/4262595311 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4005040 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d37fe700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d4113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.258+0000 7fc4d3fff700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400b3b0 tx=0x7fc4c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.259+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c400e040 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.259+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d410d720 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.259+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c400bc50 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.259+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c4013420 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.259+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d410dc60 con 0x7fc4d4102c00 2023-12-19T19:10:41.263 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.260+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d40fff10 con 0x7fc4d4102c00 2023-12-19T19:10:41.264 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.260+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc4c401c430 con 0x7fc4d4102c00 2023-12-19T19:10:41.264 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.261+0000 7fc4d0ff9700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 0x7fc4c006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:41.264 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.261+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4c4087850 con 0x7fc4d4102c00 2023-12-19T19:10:41.264 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.261+0000 7fc4d37fe700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 0x7fc4c006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:41.265 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.261+0000 7fc4d37fe700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 0x7fc4c006b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007fd0 tx=0x7fc4bc009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:41.267 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.265+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc4c4057ba0 con 0x7fc4d4102c00 2023-12-19T19:10:41.463 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.461+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4d4109c00 con 0x7fc4d4102c00 2023-12-19T19:10:41.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.461+0000 7fc4d0ff9700 1 -- 172.21.15.55:0/4262595311 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4c4057730 con 0x7fc4d4102c00 2023-12-19T19:10:41.464 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 msgr2=0x7fc4c006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 0x7fc4c006b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc007fd0 tx=0x7fc4bc009040 comp rx=0 tx=0).stop 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 msgr2=0x7fc4d4113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400b3b0 tx=0x7fc4c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 shutdown_connections 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc4c0068f60 0x7fc4c006b410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc4d4102c00 0x7fc4d4113630 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 --2- 172.21.15.55:0/4262595311 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4d41035a0 0x7fc4d4113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.463+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 >> 172.21.15.55:0/4262595311 conn(0x7fc4d40f8790 msgr2=0x7fc4d40fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.464+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 shutdown_connections 2023-12-19T19:10:41.466 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:41.464+0000 7fc4da307700 1 -- 172.21.15.55:0/4262595311 wait complete. 2023-12-19T19:10:42.077 INFO:tasks.cephadm:Setting up client nodes... 2023-12-19T19:10:42.077 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-19T19:10:42.101 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:41 smithi187 ceph-mon[87698]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:42.101 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:41 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/4262595311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T19:10:42.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:41 smithi055 ceph-mon[80978]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:42.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:41 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4262595311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T19:10:42.345 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:43.764 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:43 smithi055 ceph-mon[80978]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:44.001 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- 172.21.15.55:0/3303694568 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 msgr2=0x7fdb300ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 --2- 172.21.15.55:0/3303694568 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb300ff210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdb24004940 tx=0x7fdb24009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- 172.21.15.55:0/3303694568 shutdown_connections 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 --2- 172.21.15.55:0/3303694568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 0x7fdb301003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 --2- 172.21.15.55:0/3303694568 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb300ff210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- 172.21.15.55:0/3303694568 >> 172.21.15.55:0/3303694568 conn(0x7fdb300fa5d0 msgr2=0x7fdb300fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- 172.21.15.55:0/3303694568 shutdown_connections 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- 172.21.15.55:0/3303694568 wait complete. 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 Processor -- start 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- start start 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 0x7fdb3010fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb30110310 con 0x7fdb300fee00 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.998+0000 7fdb36e12700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb30110b50 con 0x7fdb300fff70 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:44.002 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:39024/0 (socket says 172.21.15.55:39024) 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 -- 172.21.15.55:0/2477211030 learned_addr learned my addr 172.21.15.55:0/2477211030 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb2ffff700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 0x7fdb3010fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 -- 172.21.15.55:0/2477211030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 msgr2=0x7fdb3010fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 0x7fdb3010fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 -- 172.21.15.55:0/2477211030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb24005040 con 0x7fdb300fee00 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb34bae700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009fd0 tx=0x7fdb24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2400e040 con 0x7fdb300fee00 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb30110dd0 con 0x7fdb300fee00 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb24004290 con 0x7fdb300fee00 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2401c3f0 con 0x7fdb300fee00 2023-12-19T19:10:44.003 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:43.999+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb30111310 con 0x7fdb300fee00 2023-12-19T19:10:44.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.000+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3004bb80 con 0x7fdb300fee00 2023-12-19T19:10:44.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.001+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdb24013420 con 0x7fdb300fee00 2023-12-19T19:10:44.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.001+0000 7fdb2d7fa700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 0x7fdb2006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:44.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.001+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdb24087ed0 con 0x7fdb300fee00 2023-12-19T19:10:44.005 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.001+0000 7fdb2ffff700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 0x7fdb2006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:44.005 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.002+0000 7fdb2ffff700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 0x7fdb2006b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c006fd0 tx=0x7fdb1c008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:44.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.006+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdb240580f0 con 0x7fdb300fee00 2023-12-19T19:10:44.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:43 smithi187 ceph-mon[87698]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:44.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.278+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fdb30111770 con 0x7fdb300fee00 2023-12-19T19:10:44.284 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.282+0000 7fdb2d7fa700 1 -- 172.21.15.55:0/2477211030 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fdb24057c80 con 0x7fdb300fee00 2023-12-19T19:10:44.285 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2023-12-19T19:10:44.285 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQC06oFlLKqrEBAAqFV7v92EFyNarytAS/LIIQ== 2023-12-19T19:10:44.289 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 msgr2=0x7fdb2006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:44.289 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 0x7fdb2006b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c006fd0 tx=0x7fdb1c008040 comp rx=0 tx=0).stop 2023-12-19T19:10:44.289 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 msgr2=0x7fdb3010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb24009fd0 tx=0x7fdb24009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 shutdown_connections 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdb20068f70 0x7fdb2006b420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdb300fee00 0x7fdb3010f740 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 --2- 172.21.15.55:0/2477211030 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdb300fff70 0x7fdb3010fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 >> 172.21.15.55:0/2477211030 conn(0x7fdb300fa5d0 msgr2=0x7fdb30103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 shutdown_connections 2023-12-19T19:10:44.290 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:44.284+0000 7fdb36e12700 1 -- 172.21.15.55:0/2477211030 wait complete. 2023-12-19T19:10:44.974 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:44 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2477211030' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:44.974 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:44 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2477211030' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T19:10:45.003 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:10:45.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-19T19:10:45.004 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-19T19:10:45.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-19T19:10:45.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:44 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2477211030' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:45.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:44 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2477211030' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T19:10:45.323 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi187/config 2023-12-19T19:10:45.708 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:45 smithi187 ceph-mon[87698]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:46.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:45 smithi055 ceph-mon[80978]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2381384635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 msgr2=0x7fb0ec0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2381384635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec0fd000 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc004940 tx=0x7fb0dc009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2381384635 shutdown_connections 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2381384635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2381384635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec0fd000 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:46.862 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2381384635 >> 172.21.15.187:0/2381384635 conn(0x7fb0ec0f82d0 msgr2=0x7fb0ec0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2381384635 shutdown_connections 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2381384635 wait complete. 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 Processor -- start 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- start start 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec10dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec10e0b0 con 0x7fb0ec0fde00 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.858+0000 7fb0f0cf9700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec10e950 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:43956/0 (socket says 172.21.15.187:43956) 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 -- 172.21.15.187:0/2432370196 learned_addr learned my addr 172.21.15.187:0/2432370196 (peer_addr_for_me v2:172.21.15.187:0/0) 2023-12-19T19:10:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0e1d9b700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec10dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 -- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 msgr2=0x7fb0ec10dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec10dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 -- 172.21.15.187:0/2432370196 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc005040 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0e1d9b700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec10dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0ea59c700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009360 tx=0x7fb0dc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc00e040 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0dc004290 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec10ebd0 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc01c630 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.864 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.859+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec10f110 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.860+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec1075d0 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.861+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb0dc013960 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.861+0000 7fb0e37fe700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 0x7fb0d806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.861+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0dc087cf0 con 0x7fb0ec0fcbf0 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.861+0000 7fb0e1d9b700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 0x7fb0d806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:46.865 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.862+0000 7fb0e1d9b700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 0x7fb0d806b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4007fd0 tx=0x7fb0d4009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:46.868 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:46.866+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb0dc058030 con 0x7fb0ec0fcbf0 2023-12-19T19:10:47.127 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.125+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fb0ec1075d0 con 0x7fb0ec0fcbf0 2023-12-19T19:10:47.132 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.130+0000 7fb0e37fe700 1 -- 172.21.15.187:0/2432370196 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fb0dc057bc0 con 0x7fb0ec0fcbf0 2023-12-19T19:10:47.132 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2023-12-19T19:10:47.132 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQC36oFlEa6VBxAA4q9XbPAi8e3OKr60WTPsHA== 2023-12-19T19:10:47.134 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 msgr2=0x7fb0d806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:47.134 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 0x7fb0d806b420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4007fd0 tx=0x7fb0d4009040 comp rx=0 tx=0).stop 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 msgr2=0x7fb0ec10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009360 tx=0x7fb0dc009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 shutdown_connections 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb0d8068f70 0x7fb0d806b420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0ec0fcbf0 0x7fb0ec10d570 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 --2- 172.21.15.187:0/2432370196 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb0ec0fde00 0x7fb0ec10dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 >> 172.21.15.187:0/2432370196 conn(0x7fb0ec0f82d0 msgr2=0x7fb0ec0fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 shutdown_connections 2023-12-19T19:10:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:2023-12-19T19:10:47.132+0000 7fb0f0cf9700 1 -- 172.21.15.187:0/2432370196 wait complete. 2023-12-19T19:10:47.733 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:10:47.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-19T19:10:47.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-19T19:10:47.816 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-19T19:10:47.817 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-19T19:10:47.817 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-19T19:10:48.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:47 smithi187 ceph-mon[87698]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:48.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:47 smithi187 ceph-mon[87698]: from='client.? 172.21.15.187:0/2432370196' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:48.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:47 smithi187 ceph-mon[87698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:48.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:47 smithi187 ceph-mon[87698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T19:10:48.095 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:48.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:47 smithi055 ceph-mon[80978]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:48.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:47 smithi055 ceph-mon[80978]: from='client.? 172.21.15.187:0/2432370196' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:48.218 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:47 smithi055 ceph-mon[80978]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-19T19:10:48.218 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:47 smithi055 ceph-mon[80978]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-19T19:10:49.688 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 -- 172.21.15.55:0/542128887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 msgr2=0x7fd4dc0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/542128887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc0fcc10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0004940 tx=0x7fd4d0009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 -- 172.21.15.55:0/542128887 shutdown_connections 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/542128887 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc0fcc10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/542128887 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 0x7fd4dc102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 -- 172.21.15.55:0/542128887 >> 172.21.15.55:0/542128887 conn(0x7fd4dc0f8790 msgr2=0x7fd4dc0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 -- 172.21.15.55:0/542128887 shutdown_connections 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.684+0000 7fd4e0d00700 1 -- 172.21.15.55:0/542128887 wait complete. 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 Processor -- start 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 -- start start 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 0x7fd4dc06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:49.689 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc070a80 con 0x7fd4dc1035a0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4e0d00700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4dc070bf0 con 0x7fd4dc102c00 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47368/0 (socket says 172.21.15.55:47368) 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 -- 172.21.15.55:0/4292368687 learned_addr learned my addr 172.21.15.55:0/4292368687 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4da59c700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 0x7fd4dc06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 -- 172.21.15.55:0/4292368687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 msgr2=0x7fd4dc06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 0x7fd4dc06ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.685+0000 7fd4d9d9b700 1 -- 172.21.15.55:0/4292368687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0005040 con 0x7fd4dc1035a0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4d9d9b700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0000c00 tx=0x7fd4d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4d0021070 con 0x7fd4dc1035a0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4dc1197f0 con 0x7fd4dc1035a0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4d00093b0 con 0x7fd4dc1035a0 2023-12-19T19:10:49.690 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4d000e070 con 0x7fd4dc1035a0 2023-12-19T19:10:49.691 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.686+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4dc119bf0 con 0x7fd4dc1035a0 2023-12-19T19:10:49.691 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.687+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4dc1050d0 con 0x7fd4dc1035a0 2023-12-19T19:10:49.691 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.687+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd4d001c670 con 0x7fd4dc1035a0 2023-12-19T19:10:49.692 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.688+0000 7fd4caffd700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 0x7fd4c406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:49.692 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.688+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4d0087bb0 con 0x7fd4dc1035a0 2023-12-19T19:10:49.692 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.688+0000 7fd4da59c700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 0x7fd4c406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:49.692 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.688+0000 7fd4da59c700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 0x7fd4c406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009b80 tx=0x7fd4cc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:49.694 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.692+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd4d008e080 con 0x7fd4dc1035a0 2023-12-19T19:10:49.949 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.946+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd4dc0fc390 con 0x7fd4dc1035a0 2023-12-19T19:10:49.952 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.949+0000 7fd4caffd700 1 -- 172.21.15.55:0/4292368687 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166776 (secure 0 0 0) 0x7fd4d0057e50 con 0x7fd4dc1035a0 2023-12-19T19:10:49.954 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 msgr2=0x7fd4c406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 0x7fd4c406b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009b80 tx=0x7fd4cc00b040 comp rx=0 tx=0).stop 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 msgr2=0x7fd4dc070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0000c00 tx=0x7fd4d0009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 shutdown_connections 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fd4c4068f60 0x7fd4c406b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd4dc102c00 0x7fd4dc06ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 --2- 172.21.15.55:0/4292368687 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fd4dc1035a0 0x7fd4dc070480 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.952+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 >> 172.21.15.55:0/4292368687 conn(0x7fd4dc0f8790 msgr2=0x7fd4dc0fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:49.956 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.953+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 shutdown_connections 2023-12-19T19:10:49.957 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:49.953+0000 7fd4e0d00700 1 -- 172.21.15.55:0/4292368687 wait complete. 2023-12-19T19:10:50.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:49 smithi055 ceph-mon[80978]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:50.270 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:49 smithi187 ceph-mon[87698]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:10:50.668 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":19,"active_gid":14208,"active_name":"smithi055.dwclvp","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":2515201024},{"type":"v1","addr":"172.21.15.55:6801","nonce":2515201024}]},"active_addr":"172.21.15.55:6801/2515201024","active_change":"2023-12-19T19:07:04.834230+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi187.pmhvpw","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.55:8443/","prometheus":"http://172.21.15.55:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":3555562400}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2968320296}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":250050293}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1558531773}]}]} 2023-12-19T19:10:50.672 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-19T19:10:50.672 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-19T19:10:50.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T19:10:50.943 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:51.251 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:50 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4292368687' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-19T19:10:51.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:50 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/4292368687' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-19T19:10:52.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:51 smithi055 ceph-mon[80978]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:51 smithi187 ceph-mon[87698]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 -- 172.21.15.55:0/2236228201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c64102c70 msgr2=0x7f8c64103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/2236228201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c64102c70 0x7f8c64103040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009ce0 tx=0x7f8c5c004d10 comp rx=0 tx=0).stop 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 -- 172.21.15.55:0/2236228201 shutdown_connections 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/2236228201 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c640fb1e0 0x7f8c640fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/2236228201 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c64102c70 0x7f8c64103040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 -- 172.21.15.55:0/2236228201 >> 172.21.15.55:0/2236228201 conn(0x7f8c640711a0 msgr2=0x7f8c640715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 -- 172.21.15.55:0/2236228201 shutdown_connections 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.584+0000 7f8c68a7f700 1 -- 172.21.15.55:0/2236228201 wait complete. 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 Processor -- start 2023-12-19T19:10:52.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 -- start start 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 0x7f8c64113d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c64114430 con 0x7f8c640fb1e0 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c68a7f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6410d6d0 con 0x7f8c64102c70 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c6259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c6259c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47384/0 (socket says 172.21.15.55:47384) 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.585+0000 7f8c6259c700 1 -- 172.21.15.55:0/3736084719 learned_addr learned my addr 172.21.15.55:0/3736084719 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c61d9b700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 0x7f8c64113d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c6259c700 1 -- 172.21.15.55:0/3736084719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 msgr2=0x7f8c64113d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c6259c700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 0x7f8c64113d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c6259c700 1 -- 172.21.15.55:0/3736084719 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c009940 con 0x7f8c640fb1e0 2023-12-19T19:10:52.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c61d9b700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 0x7f8c64113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c6259c700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c005010 tx=0x7f8c5c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c5c00e040 con 0x7f8c640fb1e0 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8c5c008ea0 con 0x7f8c640fb1e0 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c5c01c880 con 0x7f8c640fb1e0 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6410d950 con 0x7f8c640fb1e0 2023-12-19T19:10:52.591 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.586+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6410de90 con 0x7f8c640fb1e0 2023-12-19T19:10:52.592 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.587+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c640fff10 con 0x7f8c640fb1e0 2023-12-19T19:10:52.592 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.588+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8c5c012410 con 0x7f8c640fb1e0 2023-12-19T19:10:52.593 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.588+0000 7f8c5affd700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 0x7f8c5006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:52.593 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.588+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8c5c088690 con 0x7f8c640fb1e0 2023-12-19T19:10:52.593 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.588+0000 7f8c61d9b700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 0x7f8c5006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:52.593 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.589+0000 7f8c61d9b700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 0x7f8c5006b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c006fd0 tx=0x7f8c4c008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:52.595 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.593+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8c5c0588b0 con 0x7f8c640fb1e0 2023-12-19T19:10:52.790 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.787+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c6404bb80 con 0x7f8c640fb1e0 2023-12-19T19:10:52.790 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.788+0000 7f8c5affd700 1 -- 172.21.15.55:0/3736084719 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11091 (secure 0 0 0) 0x7f8c5c004e30 con 0x7f8c640fb1e0 2023-12-19T19:10:52.791 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:52.791 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":41,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","created":"2023-12-19T19:04:57.836701+0000","modified":"2023-12-19T19:10:33.614598+0000","last_up_change":"2023-12-19T19:10:32.592061+0000","last_in_change":"2023-12-19T19:10:13.790913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T19:09:19.364422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e4ca473b-4320-4248-b6b1-fb3182ca21ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6803","nonce":3885599653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6805","nonce":3885599653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6809","nonce":3885599653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6807","nonce":3885599653}]},"public_addr":"172.21.15.55:6803/3885599653","cluster_addr":"172.21.15.55:6805/3885599653","heartbeat_back_addr":"172.21.15.55:6809/3885599653","heartbeat_front_addr":"172.21.15.55:6807/3885599653","state":["exists","up"]},{"osd":1,"uuid":"c34d9690-2f84-4b5d-9aab-47539ed51db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6811","nonce":2714383701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6813","nonce":2714383701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6817","nonce":2714383701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6815","nonce":2714383701}]},"public_addr":"172.21.15.55:6811/2714383701","cluster_addr":"172.21.15.55:6813/2714383701","heartbeat_back_addr":"172.21.15.55:6817/2714383701","heartbeat_front_addr":"172.21.15.55:6815/2714383701","state":["exists","up"]},{"osd":2,"uuid":"a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6819","nonce":3562432537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6821","nonce":3562432537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6825","nonce":3562432537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6823","nonce":3562432537}]},"public_addr":"172.21.15.55:6819/3562432537","cluster_addr":"172.21.15.55:6821/3562432537","heartbeat_back_addr":"172.21.15.55:6825/3562432537","heartbeat_front_addr":"172.21.15.55:6823/3562432537","state":["exists","up"]},{"osd":3,"uuid":"b0ea8c34-b465-466e-a827-d096715d37f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6801","nonce":1951987134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6803","nonce":1951987134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6807","nonce":1951987134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6805","nonce":1951987134}]},"public_addr":"172.21.15.187:6801/1951987134","cluster_addr":"172.21.15.187:6803/1951987134","heartbeat_back_addr":"172.21.15.187:6807/1951987134","heartbeat_front_addr":"172.21.15.187:6805/1951987134","state":["exists","up"]},{"osd":4,"uuid":"d6d80173-9176-4bde-a702-f4a43309d5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6809","nonce":2860720480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6811","nonce":2860720480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6815","nonce":2860720480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6813","nonce":2860720480}]},"public_addr":"172.21.15.187:6809/2860720480","cluster_addr":"172.21.15.187:6811/2860720480","heartbeat_back_addr":"172.21.15.187:6815/2860720480","heartbeat_front_addr":"172.21.15.187:6813/2860720480","state":["exists","up"]},{"osd":5,"uuid":"dc051ac7-0460-497e-a684-63aea173b125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6817","nonce":1722753525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6819","nonce":1722753525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6823","nonce":1722753525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6821","nonce":1722753525}]},"public_addr":"172.21.15.187:6817/1722753525","cluster_addr":"172.21.15.187:6819/1722753525","heartbeat_back_addr":"172.21.15.187:6823/1722753525","heartbeat_front_addr":"172.21.15.187:6821/1722753525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:20.662263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:47.762312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:16.369465+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:39.397498+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:05.487467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:30.696534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/1287666552":"2023-12-20T19:07:04.834074+0000","172.21.15.55:6800/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2442422185":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/3109624574":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/907073539":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2629565350":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6800/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/937693294":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/2933015314":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/4084383373":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2133728457":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6800/917343204":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6801/917343204":"2023-12-20T19:05:32.358078+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 msgr2=0x7f8c5006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 0x7f8c5006b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c006fd0 tx=0x7f8c4c008040 comp rx=0 tx=0).stop 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 msgr2=0x7f8c64113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c005010 tx=0x7f8c5c004d10 comp rx=0 tx=0).stop 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 shutdown_connections 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8c50068f70 0x7f8c5006b420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8c640fb1e0 0x7f8c64113810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 --2- 172.21.15.55:0/3736084719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8c64102c70 0x7f8c64113d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 >> 172.21.15.55:0/3736084719 conn(0x7f8c640711a0 msgr2=0x7f8c640fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:52.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 shutdown_connections 2023-12-19T19:10:52.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:52.790+0000 7f8c68a7f700 1 -- 172.21.15.55:0/3736084719 wait complete. 2023-12-19T19:10:53.542 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-19T19:10:53.543 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-19T19:10:53.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:53 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3736084719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:53.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:53 smithi055 ceph-mon[80978]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:53.814 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:54.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:53 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3736084719' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:54.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:53 smithi187 ceph-mon[87698]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:55.341 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 -- 172.21.15.55:0/2568387064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c101b00 msgr2=0x7f6b0c101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 --2- 172.21.15.55:0/2568387064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c101b00 0x7f6b0c101ed0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6b08004940 tx=0x7f6b08009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 -- 172.21.15.55:0/2568387064 shutdown_connections 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 --2- 172.21.15.55:0/2568387064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c0fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 --2- 172.21.15.55:0/2568387064 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c101b00 0x7f6b0c101ed0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 -- 172.21.15.55:0/2568387064 >> 172.21.15.55:0/2568387064 conn(0x7f6b0c0719b0 msgr2=0x7f6b0c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.337+0000 7f6b140b7700 1 -- 172.21.15.55:0/2568387064 shutdown_connections 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 -- 172.21.15.55:0/2568387064 wait complete. 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 Processor -- start 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 -- start start 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c101b00 0x7f6b0c113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c114220 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b140b7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c10d510 con 0x7f6b0c101b00 2023-12-19T19:10:55.342 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b11e53700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b11e53700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:47414/0 (socket says 172.21.15.55:47414) 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b11e53700 1 -- 172.21.15.55:0/938022835 learned_addr learned my addr 172.21.15.55:0/938022835 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.338+0000 7f6b11652700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c101b00 0x7f6b0c113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b11e53700 1 -- 172.21.15.55:0/938022835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c101b00 msgr2=0x7f6b0c113bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b11e53700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c101b00 0x7f6b0c113bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b11e53700 1 -- 172.21.15.55:0/938022835 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08005040 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b11e53700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009360 tx=0x7f6b08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b0800e040 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b08013420 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c10d740 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b0801c610 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.343 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.339+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c10dc80 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.340+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c0734d0 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.341+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6b08013590 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.341+0000 7f6b027fc700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 0x7f6af806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.341+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6b08087cf0 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.341+0000 7f6b11652700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 0x7f6af806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:55.344 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.342+0000 7f6b11652700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 0x7f6af806b420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c10eef0 tx=0x7f6afc0043c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:55.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.346+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6b08057fc0 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.546 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.544+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b0c0734d0 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.547 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.545+0000 7f6b027fc700 1 -- 172.21.15.55:0/938022835 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11091 (secure 0 0 0) 0x7f6b08057b50 con 0x7f6b0c0fbb40 2023-12-19T19:10:55.547 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:10:55.547 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":41,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","created":"2023-12-19T19:04:57.836701+0000","modified":"2023-12-19T19:10:33.614598+0000","last_up_change":"2023-12-19T19:10:32.592061+0000","last_in_change":"2023-12-19T19:10:13.790913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T19:09:19.364422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e4ca473b-4320-4248-b6b1-fb3182ca21ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6803","nonce":3885599653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6805","nonce":3885599653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6809","nonce":3885599653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6807","nonce":3885599653}]},"public_addr":"172.21.15.55:6803/3885599653","cluster_addr":"172.21.15.55:6805/3885599653","heartbeat_back_addr":"172.21.15.55:6809/3885599653","heartbeat_front_addr":"172.21.15.55:6807/3885599653","state":["exists","up"]},{"osd":1,"uuid":"c34d9690-2f84-4b5d-9aab-47539ed51db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6811","nonce":2714383701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6813","nonce":2714383701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6817","nonce":2714383701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6815","nonce":2714383701}]},"public_addr":"172.21.15.55:6811/2714383701","cluster_addr":"172.21.15.55:6813/2714383701","heartbeat_back_addr":"172.21.15.55:6817/2714383701","heartbeat_front_addr":"172.21.15.55:6815/2714383701","state":["exists","up"]},{"osd":2,"uuid":"a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6819","nonce":3562432537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6821","nonce":3562432537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6825","nonce":3562432537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6823","nonce":3562432537}]},"public_addr":"172.21.15.55:6819/3562432537","cluster_addr":"172.21.15.55:6821/3562432537","heartbeat_back_addr":"172.21.15.55:6825/3562432537","heartbeat_front_addr":"172.21.15.55:6823/3562432537","state":["exists","up"]},{"osd":3,"uuid":"b0ea8c34-b465-466e-a827-d096715d37f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6801","nonce":1951987134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6803","nonce":1951987134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6807","nonce":1951987134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6805","nonce":1951987134}]},"public_addr":"172.21.15.187:6801/1951987134","cluster_addr":"172.21.15.187:6803/1951987134","heartbeat_back_addr":"172.21.15.187:6807/1951987134","heartbeat_front_addr":"172.21.15.187:6805/1951987134","state":["exists","up"]},{"osd":4,"uuid":"d6d80173-9176-4bde-a702-f4a43309d5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6809","nonce":2860720480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6811","nonce":2860720480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6815","nonce":2860720480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6813","nonce":2860720480}]},"public_addr":"172.21.15.187:6809/2860720480","cluster_addr":"172.21.15.187:6811/2860720480","heartbeat_back_addr":"172.21.15.187:6815/2860720480","heartbeat_front_addr":"172.21.15.187:6813/2860720480","state":["exists","up"]},{"osd":5,"uuid":"dc051ac7-0460-497e-a684-63aea173b125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6817","nonce":1722753525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6819","nonce":1722753525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6823","nonce":1722753525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6821","nonce":1722753525}]},"public_addr":"172.21.15.187:6817/1722753525","cluster_addr":"172.21.15.187:6819/1722753525","heartbeat_back_addr":"172.21.15.187:6823/1722753525","heartbeat_front_addr":"172.21.15.187:6821/1722753525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:20.662263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:47.762312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:16.369465+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:39.397498+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:05.487467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:30.696534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/1287666552":"2023-12-20T19:07:04.834074+0000","172.21.15.55:6800/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2442422185":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/3109624574":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/907073539":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2629565350":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6800/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/937693294":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/2933015314":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/4084383373":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2133728457":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6800/917343204":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6801/917343204":"2023-12-20T19:05:32.358078+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 msgr2=0x7f6af806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 0x7f6af806b420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c10eef0 tx=0x7f6afc0043c0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 msgr2=0x7f6b0c113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009360 tx=0x7f6b08009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 shutdown_connections 2023-12-19T19:10:55.550 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f6af8068f70 0x7f6af806b420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.551 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f6b0c0fbb40 0x7f6b0c113650 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.551 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 --2- 172.21.15.55:0/938022835 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6b0c101b00 0x7f6b0c113bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:55.551 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 >> 172.21.15.55:0/938022835 conn(0x7f6b0c0719b0 msgr2=0x7f6b0c065490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:55.551 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 shutdown_connections 2023-12-19T19:10:55.551 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:55.547+0000 7f6b140b7700 1 -- 172.21.15.55:0/938022835 wait complete. 2023-12-19T19:10:56.218 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:55 smithi055 ceph-mon[80978]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:56.218 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:55 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/938022835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:56.244 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-19T19:10:56.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-19T19:10:56.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-19T19:10:56.245 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-19T19:10:56.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-19T19:10:56.246 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-19T19:10:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:55 smithi187 ceph-mon[87698]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:55 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/938022835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:10:56.563 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:56.565 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:58.153 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:57 smithi055 ceph-mon[80978]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:58.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:57 smithi187 ceph-mon[87698]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:10:58.475 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:58.476 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:10:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 -- 172.21.15.55:0/1867937903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 msgr2=0x7fa6b00fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/1867937903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b00fcc10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4004940 tx=0x7fa6a4009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 -- 172.21.15.55:0/1867937903 shutdown_connections 2023-12-19T19:10:59.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/1867937903 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b00fcc10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/1867937903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6b0102c00 0x7fa6b0102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 -- 172.21.15.55:0/1867937903 >> 172.21.15.55:0/1867937903 conn(0x7fa6b00f8790 msgr2=0x7fa6b00faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 -- 172.21.15.55:0/1867937903 shutdown_connections 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.140+0000 7fa6b4d65700 1 -- 172.21.15.55:0/1867937903 wait complete. 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 Processor -- start 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 -- start start 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6b0102c00 0x7fa6b006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0070b10 con 0x7fa6b01035a0 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6b4d65700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0070c80 con 0x7fa6b0102c00 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6add9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6add9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49338/0 (socket says 172.21.15.55:49338) 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.141+0000 7fa6add9b700 1 -- 172.21.15.55:0/2238079757 learned_addr learned my addr 172.21.15.55:0/2238079757 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:10:59.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6add9b700 1 -- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6b0102c00 msgr2=0x7fa6b006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6add9b700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6b0102c00 0x7fa6b006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6add9b700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a4005040 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6add9b700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009180 tx=0x7fa6a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a400e040 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6a4004290 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b01198a0 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa6a401c3f0 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.142+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0119d90 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.143+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6b005e380 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.143+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa6a400bc50 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa69effd700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 0x7fa69806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa6a4087d60 con 0x7fa6b01035a0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa6ae59c700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 0x7fa69806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:59.154 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa69effd700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 0x7fa698071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6980717d0 con 0x7fa69806ed10 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a408d270 con 0x7fa6b01035a0 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa6ae59c700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 0x7fa69806b4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0006fd0 tx=0x7fa6a0006db0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.144+0000 7fa6aed9d700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 0x7fa698071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.145+0000 7fa6aed9d700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 0x7fa698071120 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:10:59.155 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.146+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== osd.3 v2:172.21.15.187:6800/1951987134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa6980717d0 con 0x7fa69806ed10 2023-12-19T19:10:59.164 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.162+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 --> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6b005e7b0 con 0x7fa69806ed10 2023-12-19T19:10:59.164 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.162+0000 7fa69effd700 1 -- 172.21.15.55:0/2238079757 <== osd.3 v2:172.21.15.187:6800/1951987134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa6b005e7b0 con 0x7fa69806ed10 2023-12-19T19:10:59.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.162+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 msgr2=0x7fa698071120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:59.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.162+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 0x7fa698071120 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 msgr2=0x7fa69806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:59.165 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 0x7fa69806b4f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0006fd0 tx=0x7fa6a0006db0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 msgr2=0x7fa6b0070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009180 tx=0x7fa6a4009e30 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 shutdown_connections 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:6800/1951987134,v1:172.21.15.187:6801/1951987134] conn(0x7fa69806ed10 0x7fa698071120 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa698069040 0x7fa69806b4f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6b0102c00 0x7fa6b006ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 --2- 172.21.15.55:0/2238079757 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa6b01035a0 0x7fa6b0070480 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 >> 172.21.15.55:0/2238079757 conn(0x7fa6b00f8790 msgr2=0x7fa6b00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 shutdown_connections 2023-12-19T19:10:59.166 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:10:59.163+0000 7fa6b4d65700 1 -- 172.21.15.55:0/2238079757 wait complete. 2023-12-19T19:11:00.167 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:10:59 smithi055 ceph-mon[80978]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:00.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:10:59 smithi187 ceph-mon[87698]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:00.750 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 -- 172.21.15.55:0/457154360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 msgr2=0x7fdbf8108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/457154360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8108fe0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8004940 tx=0x7fdbe8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 -- 172.21.15.55:0/457154360 shutdown_connections 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/457154360 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8108fe0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/457154360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 0x7fdbf80fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 -- 172.21.15.55:0/457154360 >> 172.21.15.55:0/457154360 conn(0x7fdbf80f76c0 msgr2=0x7fdbf80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 -- 172.21.15.55:0/457154360 shutdown_connections 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 -- 172.21.15.55:0/457154360 wait complete. 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.748+0000 7fdbff7f1700 1 Processor -- start 2023-12-19T19:11:00.751 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbff7f1700 1 -- start start 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbff7f1700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 0x7fdbf8115820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbff7f1700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbff7f1700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf81163b0 con 0x7fdbf80fc280 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbff7f1700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf810f6e0 con 0x7fdbf80fb970 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49352/0 (socket says 172.21.15.55:49352) 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 -- 172.21.15.55:0/3656386413 learned_addr learned my addr 172.21.15.55:0/3656386413 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfd58d700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 0x7fdbf8115820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:00.753 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 -- 172.21.15.55:0/3656386413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 msgr2=0x7fdbf8115820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 0x7fdbf8115820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe8005040 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.749+0000 7fdbfcd8c700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800b3b0 tx=0x7fdbe8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.750+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe800e040 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.750+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbe800bc50 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.750+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf810f960 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.750+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbe801c630 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.750+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf810fea0 con 0x7fdbf80fc280 2023-12-19T19:11:00.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.751+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdbf81103c0 con 0x7fdbf80fc280 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.751+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdbe8013960 con 0x7fdbf80fc280 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbedffb700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 0x7fdbe406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdbe8087f10 con 0x7fdbf80fc280 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbfd58d700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 0x7fdbe406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbedffb700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 0x7fdbe4071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbe40716f0 con 0x7fdbe406ec30 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbe808d510 con 0x7fdbf80fc280 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.752+0000 7fdbfdd8e700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 0x7fdbe4071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.753+0000 7fdbfd58d700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 0x7fdbe406b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4004990 tx=0x7fdbf4009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:00.756 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.753+0000 7fdbfdd8e700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 0x7fdbe4071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:00.757 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.753+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== osd.1 v2:172.21.15.55:6810/2714383701 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fdbe40716f0 con 0x7fdbe406ec30 2023-12-19T19:11:00.772 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.769+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 --> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdbf8110580 con 0x7fdbe406ec30 2023-12-19T19:11:00.772 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.769+0000 7fdbedffb700 1 -- 172.21.15.55:0/3656386413 <== osd.1 v2:172.21.15.55:6810/2714383701 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdbf8110580 con 0x7fdbe406ec30 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 msgr2=0x7fdbe4071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 0x7fdbe4071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 msgr2=0x7fdbe406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 0x7fdbe406b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4004990 tx=0x7fdbf4009ee0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 msgr2=0x7fdbf8115d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800b3b0 tx=0x7fdbe8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 shutdown_connections 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6810/2714383701,v1:172.21.15.55:6811/2714383701] conn(0x7fdbe406ec30 0x7fdbe4071040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fdbe4068f60 0x7fdbe406b410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdbf80fb970 0x7fdbf8115820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 --2- 172.21.15.55:0/3656386413 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fdbf80fc280 0x7fdbf8115d60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 >> 172.21.15.55:0/3656386413 conn(0x7fdbf80f76c0 msgr2=0x7fdbf80f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:00.773 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 shutdown_connections 2023-12-19T19:11:00.774 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:00.770+0000 7fdbff7f1700 1 -- 172.21.15.55:0/3656386413 wait complete. 2023-12-19T19:11:01.987 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:01.987 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:02.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:01 smithi055 ceph-mon[80978]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:01 smithi187 ceph-mon[87698]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:02.393 INFO:teuthology.orchestra.run.smithi055.stdout:115964117011 2023-12-19T19:11:02.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-19T19:11:03.011 INFO:teuthology.orchestra.run.smithi055.stdout:60129542174 2023-12-19T19:11:03.012 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-19T19:11:03.698 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.695+0000 7f859ac0e700 1 -- 172.21.15.55:0/1698697150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594102c20 msgr2=0x7f8594102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.698 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.695+0000 7f859ac0e700 1 --2- 172.21.15.55:0/1698697150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594102c20 0x7f8594102ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8584009ce0 tx=0x7f8584004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 -- 172.21.15.55:0/1698697150 shutdown_connections 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 --2- 172.21.15.55:0/1698697150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594065390 0x7f8594065800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 --2- 172.21.15.55:0/1698697150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594102c20 0x7f8594102ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 -- 172.21.15.55:0/1698697150 >> 172.21.15.55:0/1698697150 conn(0x7f85940719b0 msgr2=0x7f8594071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 -- 172.21.15.55:0/1698697150 shutdown_connections 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 -- 172.21.15.55:0/1698697150 wait complete. 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 Processor -- start 2023-12-19T19:11:03.699 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.696+0000 7f859ac0e700 1 -- start start 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f859ac0e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f859ac0e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594102c20 0x7f8594110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f859ac0e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85941111a0 con 0x7f8594065390 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f859ac0e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85941163f0 con 0x7f8594102c20 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49376/0 (socket says 172.21.15.55:49376) 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 -- 172.21.15.55:0/55568608 learned_addr learned my addr 172.21.15.55:0/55568608 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:03.701 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f8593fff700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594102c20 0x7f8594110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 -- 172.21.15.55:0/55568608 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594102c20 msgr2=0x7f8594110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594102c20 0x7f8594110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8584009940 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85989aa700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f858400bee0 tx=0x7f8584004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858400e040 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.697+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8594116670 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.698+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8584008f00 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.698+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8594116bb0 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.698+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f858401c710 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.698+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8574001000 con 0x7f8594065390 2023-12-19T19:11:03.702 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.699+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8584012410 con 0x7f8594065390 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.699+0000 7f85917fa700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 0x7f858006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.699+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8584088570 con 0x7f8594065390 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.699+0000 7f8593fff700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 0x7f858006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f85917fa700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 0x7f8580071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85800716f0 con 0x7f858006ec30 2023-12-19T19:11:03.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f858408abc0 con 0x7f8594065390 2023-12-19T19:11:03.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f85991ab700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 0x7f8580071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f85991ab700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 0x7f8580071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.700+0000 7f8593fff700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 0x7f858006b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f857c007fd0 tx=0x7f857c009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.701+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== osd.2 v2:172.21.15.55:6818/3562432537 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f85800716f0 con 0x7f858006ec30 2023-12-19T19:11:03.718 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.716+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 --> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8574002d60 con 0x7f858006ec30 2023-12-19T19:11:03.718 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.716+0000 7f85917fa700 1 -- 172.21.15.55:0/55568608 <== osd.2 v2:172.21.15.55:6818/3562432537 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8574002d60 con 0x7f858006ec30 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.716+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 msgr2=0x7f8580071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 0x7f8580071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 msgr2=0x7f858006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 0x7f858006b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f857c007fd0 tx=0x7f857c009040 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 msgr2=0x7f8594110610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f858400bee0 tx=0x7f8584004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 shutdown_connections 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8580068f60 0x7f858006b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:6818/3562432537,v1:172.21.15.55:6819/3562432537] conn(0x7f858006ec30 0x7f8580071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8594065390 0x7f8594110610 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 --2- 172.21.15.55:0/55568608 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8594102c20 0x7f8594110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 >> 172.21.15.55:0/55568608 conn(0x7f85940719b0 msgr2=0x7f85940fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:03.720 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 shutdown_connections 2023-12-19T19:11:03.721 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.717+0000 7f859ac0e700 1 -- 172.21.15.55:0/55568608 wait complete. 2023-12-19T19:11:03.828 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:03 smithi055 ceph-mon[80978]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:03.895 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1823361834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc0fbb40 msgr2=0x7f4bcc0fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1823361834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc0fbfb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004940 tx=0x7f4bbc009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1823361834 shutdown_connections 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1823361834 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc0fbfb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1823361834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc101b00 0x7f4bcc101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1823361834 >> 172.21.15.55:0/1823361834 conn(0x7f4bcc0f76c0 msgr2=0x7f4bcc0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1823361834 shutdown_connections 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.873+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1823361834 wait complete. 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 Processor -- start 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 -- start start 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc10e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc10f1d0 con 0x7f4bcc101b00 2023-12-19T19:11:03.896 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bd13b8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc10f310 con 0x7f4bcc0fbb40 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bca7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.874+0000 7f4bca7fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49394/0 (socket says 172.21.15.55:49394) 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bca7fc700 1 -- 172.21.15.55:0/1125420503 learned_addr learned my addr 172.21.15.55:0/1125420503 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bcaffd700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc10e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bca7fc700 1 -- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 msgr2=0x7f4bcc10e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bca7fc700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc10e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bca7fc700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc005040 con 0x7f4bcc101b00 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bcaffd700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bca7fc700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00b3b0 tx=0x7f4bbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc00e040 con 0x7f4bcc101b00 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc117610 con 0x7f4bcc101b00 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bbc00bc50 con 0x7f4bcc101b00 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4bbc013420 con 0x7f4bcc101b00 2023-12-19T19:11:03.897 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.875+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc117b20 con 0x7f4bcc101b00 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.876+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4bcc05e380 con 0x7f4bcc101b00 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4bbc013690 con 0x7f4bcc101b00 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 0x7f4bb806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4bbc0877e0 con 0x7f4bcc101b00 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bcaffd700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 0x7f4bb806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 0x7f4bb8071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bb80716f0 con 0x7f4bb806ec30 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.877+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4bbc08c220 con 0x7f4bcc101b00 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.878+0000 7f4bcb7fe700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 0x7f4bb8071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.878+0000 7f4bcaffd700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 0x7f4bb806b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4007fd0 tx=0x7f4bb4009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.878+0000 7f4bcb7fe700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 0x7f4bb8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:03.898 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.879+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== osd.4 v2:172.21.15.187:6808/2860720480 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4bb80716f0 con 0x7f4bb806ec30 2023-12-19T19:11:03.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.894+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 --> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4bcc05e7b0 con 0x7f4bb806ec30 2023-12-19T19:11:03.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bc37fe700 1 -- 172.21.15.55:0/1125420503 <== osd.4 v2:172.21.15.187:6808/2860720480 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4bcc05e7b0 con 0x7f4bb806ec30 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 msgr2=0x7f4bb8071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 0x7f4bb8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 msgr2=0x7f4bb806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 0x7f4bb806b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4007fd0 tx=0x7f4bb4009040 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 msgr2=0x7f4bcc10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00b3b0 tx=0x7f4bbc009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 shutdown_connections 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f4bb8068f60 0x7f4bb806b410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:6808/2860720480,v1:172.21.15.187:6809/2860720480] conn(0x7f4bb806ec30 0x7f4bb8071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bcc0fbb40 0x7f4bcc10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 --2- 172.21.15.55:0/1125420503 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f4bcc101b00 0x7f4bcc10ec40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:03.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.895+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 >> 172.21.15.55:0/1125420503 conn(0x7f4bcc0f76c0 msgr2=0x7f4bcc0f9350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:03.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.896+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 shutdown_connections 2023-12-19T19:11:03.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:03.896+0000 7f4bd13b8700 1 -- 172.21.15.55:0/1125420503 wait complete. 2023-12-19T19:11:04.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:03 smithi187 ceph-mon[87698]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:04.338 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:04.767 INFO:teuthology.orchestra.run.smithi055.stdout:150323855373 2023-12-19T19:11:04.768 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-19T19:11:04.987 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:05.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:06.192 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:05 smithi055 ceph-mon[80978]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:06.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:05 smithi187 ceph-mon[87698]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:07.007 INFO:teuthology.orchestra.run.smithi055.stdout:81604378648 2023-12-19T19:11:07.008 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 -- 172.21.15.55:0/384183004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 msgr2=0x7f203c065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 --2- 172.21.15.55:0/384183004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c065800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f202c004940 tx=0x7f202c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 -- 172.21.15.55:0/384183004 shutdown_connections 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 --2- 172.21.15.55:0/384183004 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c065800 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 --2- 172.21.15.55:0/384183004 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 0x7f203c102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.446 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 -- 172.21.15.55:0/384183004 >> 172.21.15.55:0/384183004 conn(0x7f203c0719b0 msgr2=0x7f203c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 -- 172.21.15.55:0/384183004 shutdown_connections 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 -- 172.21.15.55:0/384183004 wait complete. 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.439+0000 7f2043dad700 1 Processor -- start 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2043dad700 1 -- start start 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2043dad700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2043dad700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 0x7f203c113c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2043dad700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10d570 con 0x7f203c065390 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2043dad700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c10d6e0 con 0x7f203c102c20 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49418/0 (socket says 172.21.15.55:49418) 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 -- 172.21.15.55:0/1543671045 learned_addr learned my addr 172.21.15.55:0/1543671045 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041348700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 0x7f203c113c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 -- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 msgr2=0x7f203c113c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:07.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 0x7f203c113c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c005040 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.440+0000 7f2041b49700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f203800b790 tx=0x7f2038009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203800f770 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c10d9c0 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f203800fd80 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f203800d310 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c10df00 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.441+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f203c10e400 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.442+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2038010420 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.442+0000 7f20327fc700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 0x7f202806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2038086a40 con 0x7f203c065390 2023-12-19T19:11:07.448 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f2041348700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 0x7f202806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f20327fc700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 0x7f2028071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20280716f0 con 0x7f202806ec30 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f203808b220 con 0x7f203c065390 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f204234a700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 0x7f2028071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.443+0000 7f2041348700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 0x7f202806b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f202c004f90 tx=0x7f202c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:07.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.444+0000 7f204234a700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 0x7f2028071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:07.450 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.444+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== osd.5 v2:172.21.15.187:6816/1722753525 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f20280716f0 con 0x7f202806ec30 2023-12-19T19:11:07.462 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.460+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 --> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f203c05e2c0 con 0x7f202806ec30 2023-12-19T19:11:07.463 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.460+0000 7f20327fc700 1 -- 172.21.15.55:0/1543671045 <== osd.5 v2:172.21.15.187:6816/1722753525 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f203c05e2c0 con 0x7f202806ec30 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 msgr2=0x7f2028071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 0x7f2028071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 msgr2=0x7f202806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 0x7f202806b410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f202c004f90 tx=0x7f202c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 msgr2=0x7f203c1136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f203800b790 tx=0x7f2038009250 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 shutdown_connections 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:6816/1722753525,v1:172.21.15.187:6817/1722753525] conn(0x7f202806ec30 0x7f2028071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2028068f60 0x7f202806b410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f203c065390 0x7f203c1136b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 --2- 172.21.15.55:0/1543671045 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f203c102c20 0x7f203c113c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 >> 172.21.15.55:0/1543671045 conn(0x7f203c0719b0 msgr2=0x7f203c0fab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:07.464 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 shutdown_connections 2023-12-19T19:11:07.465 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:07.461+0000 7f2043dad700 1 -- 172.21.15.55:0/1543671045 wait complete. 2023-12-19T19:11:08.237 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:07 smithi055 ceph-mon[80978]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:07 smithi187 ceph-mon[87698]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:09.340 INFO:teuthology.orchestra.run.smithi055.stdout:171798691848 2023-12-19T19:11:09.341 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-19T19:11:09.918 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:10.185 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:09 smithi055 ceph-mon[80978]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:09 smithi187 ceph-mon[87698]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:10.766 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.762+0000 7ff2b4e26700 1 -- 172.21.15.55:0/2033678749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fdc10 msgr2=0x7ff2b00fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:10.766 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.762+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/2033678749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fdc10 0x7ff2b00fdfe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0004940 tx=0x7ff2a0009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:10.766 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 -- 172.21.15.55:0/2033678749 shutdown_connections 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/2033678749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/2033678749 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fdc10 0x7ff2b00fdfe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 -- 172.21.15.55:0/2033678749 >> 172.21.15.55:0/2033678749 conn(0x7ff2b00f76c0 msgr2=0x7ff2b00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 -- 172.21.15.55:0/2033678749 shutdown_connections 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 -- 172.21.15.55:0/2033678749 wait complete. 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 Processor -- start 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.763+0000 7ff2b4e26700 1 -- start start 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2b4e26700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fdc10 0x7ff2b01135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2b4e26700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2b4e26700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0114140 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2b4e26700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b010d470 con 0x7ff2b00fdc10 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2add9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2add9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40692/0 (socket says 172.21.15.55:40692) 2023-12-19T19:11:10.767 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2add9b700 1 -- 172.21.15.55:0/4011771033 learned_addr learned my addr 172.21.15.55:0/4011771033 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:10.768 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.764+0000 7ff2ae59c700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fdc10 0x7ff2b01135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:10.768 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2add9b700 1 -- 172.21.15.55:0/4011771033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fdc10 msgr2=0x7ff2b01135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:10.768 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2add9b700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fdc10 0x7ff2b01135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.768 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2add9b700 1 -- 172.21.15.55:0/4011771033 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a0005040 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2add9b700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400eb40 tx=0x7ff2a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a4007cc0 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2a400f040 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2a400c990 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b010d610 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.765+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b010da90 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.767+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff2a400caf0 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.769 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.767+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b0104200 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.770 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.768+0000 7ff29effd700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 0x7ff29806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:10.770 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.768+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff2a4086630 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.770 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.768+0000 7ff2ae59c700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 0x7ff29806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:10.771 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.769+0000 7ff2ae59c700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 0x7ff29806b4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000b0d0 tx=0x7ff2a0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:10.775 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.773+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff2a404da00 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.805 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:10.806 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:10.984 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.981+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff2b00fb2c0 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.984 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.981+0000 7ff29effd700 1 -- 172.21.15.55:0/4011771033 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff2a40568e0 con 0x7ff2b00fe5b0 2023-12-19T19:11:10.984 INFO:teuthology.orchestra.run.smithi055.stdout:115964117013 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 msgr2=0x7ff29806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 0x7ff29806b4e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff2a000b0d0 tx=0x7ff2a0009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 msgr2=0x7ff2b0113af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400eb40 tx=0x7ff2a400b040 comp rx=0 tx=0).stop 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 shutdown_connections 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff298069030 0x7ff29806b4e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2b00fdc10 0x7ff2b01135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 --2- 172.21.15.55:0/4011771033 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff2b00fe5b0 0x7ff2b0113af0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 >> 172.21.15.55:0/4011771033 conn(0x7ff2b00f76c0 msgr2=0x7ff2b0101380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:10.986 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 shutdown_connections 2023-12-19T19:11:10.987 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:10.983+0000 7ff2b4e26700 1 -- 172.21.15.55:0/4011771033 wait complete. 2023-12-19T19:11:11.760 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117011 got 115964117013 for osd.3 2023-12-19T19:11:11.760 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:12.018 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:11 smithi055 ceph-mon[80978]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:12.019 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:11 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4011771033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 -- 172.21.15.55:0/3354436098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 msgr2=0x7fa9dc0fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 --2- 172.21.15.55:0/3354436098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc0fb680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4004940 tx=0x7fa9c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 -- 172.21.15.55:0/3354436098 shutdown_connections 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 --2- 172.21.15.55:0/3354436098 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc0fb680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 --2- 172.21.15.55:0/3354436098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.230 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 -- 172.21.15.55:0/3354436098 >> 172.21.15.55:0/3354436098 conn(0x7fa9dc0711b0 msgr2=0x7fa9dc0715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:12.231 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 -- 172.21.15.55:0/3354436098 shutdown_connections 2023-12-19T19:11:12.231 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.227+0000 7fa9e173b700 1 -- 172.21.15.55:0/3354436098 wait complete. 2023-12-19T19:11:12.232 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.228+0000 7fa9e173b700 1 Processor -- start 2023-12-19T19:11:12.232 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.228+0000 7fa9e173b700 1 -- start start 2023-12-19T19:11:12.232 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9e173b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9e173b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9e173b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc114200 con 0x7fa9dc0fb210 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9e173b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc10d4f0 con 0x7fa9dc102c90 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40718/0 (socket says 172.21.15.55:40718) 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 -- 172.21.15.55:0/2022032092 learned_addr learned my addr 172.21.15.55:0/2022032092 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9da7fc700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 -- 172.21.15.55:0/2022032092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 msgr2=0x7fa9dc113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4005040 con 0x7fa9dc0fb210 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9da7fc700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9daffd700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400c8b0 tx=0x7fa9d4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:12.233 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d400a760 con 0x7fa9dc0fb210 2023-12-19T19:11:12.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc10d780 con 0x7fa9dc0fb210 2023-12-19T19:11:12.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.229+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9d400ad70 con 0x7fa9dc0fb210 2023-12-19T19:11:12.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.230+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9d40113a0 con 0x7fa9dc0fb210 2023-12-19T19:11:12.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.230+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc10dcc0 con 0x7fa9dc0fb210 2023-12-19T19:11:12.234 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.230+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa9dc10e210 con 0x7fa9dc0fb210 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.231+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa9d4012400 con 0x7fa9dc0fb210 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.231+0000 7fa9d37fe700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 0x7fa9c806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9da7fc700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 0x7fa9c806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9d4087630 con 0x7fa9dc0fb210 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9d37fe700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 0x7fa9c8071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9c80716f0 con 0x7fa9c806ec30 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9d4087a30 con 0x7fa9dc0fb210 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9db7fe700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 0x7fa9c8071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9da7fc700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 0x7fa9c806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000c00 tx=0x7fa9c4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:12.235 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.232+0000 7fa9db7fe700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 0x7fa9c8071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:12.236 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.233+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== osd.0 v2:172.21.15.55:6802/3885599653 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa9c80716f0 con 0x7fa9c806ec30 2023-12-19T19:11:12.251 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.248+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 --> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa9dc05e2c0 con 0x7fa9c806ec30 2023-12-19T19:11:12.251 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9d37fe700 1 -- 172.21.15.55:0/2022032092 <== osd.0 v2:172.21.15.55:6802/3885599653 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa9dc05e2c0 con 0x7fa9c806ec30 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 msgr2=0x7fa9c8071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 0x7fa9c8071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 msgr2=0x7fa9c806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 0x7fa9c806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4000c00 tx=0x7fa9c4009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 msgr2=0x7fa9dc113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400c8b0 tx=0x7fa9d4004470 comp rx=0 tx=0).stop 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 shutdown_connections 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6802/3885599653,v1:172.21.15.55:6803/3885599653] conn(0x7fa9c806ec30 0x7fa9c8071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa9c8068f60 0x7fa9c806b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa9dc0fb210 0x7fa9dc113630 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 --2- 172.21.15.55:0/2022032092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9dc102c90 0x7fa9dc113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 >> 172.21.15.55:0/2022032092 conn(0x7fa9dc0711b0 msgr2=0x7fa9dc0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 shutdown_connections 2023-12-19T19:11:12.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:12.249+0000 7fa9e173b700 1 -- 172.21.15.55:0/2022032092 wait complete. 2023-12-19T19:11:12.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:11 smithi187 ceph-mon[87698]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:12.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:11 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/4011771033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-19T19:11:13.782 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:13 smithi055 ceph-mon[80978]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:14.022 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:13 smithi187 ceph-mon[87698]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:14.745 INFO:teuthology.orchestra.run.smithi055.stdout:38654705700 2023-12-19T19:11:14.746 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 -- 172.21.15.55:0/1399931966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c8102c70 msgr2=0x7fa3c8103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/1399931966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c8102c70 0x7fa3c8103040 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8004940 tx=0x7fa3b8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 -- 172.21.15.55:0/1399931966 shutdown_connections 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/1399931966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c80fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/1399931966 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c8102c70 0x7fa3c8103040 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 -- 172.21.15.55:0/1399931966 >> 172.21.15.55:0/1399931966 conn(0x7fa3c80711a0 msgr2=0x7fa3c80715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 -- 172.21.15.55:0/1399931966 shutdown_connections 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.276+0000 7fa3cddf7700 1 -- 172.21.15.55:0/1399931966 wait complete. 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 Processor -- start 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 -- start start 2023-12-19T19:11:15.280 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c8102c70 0x7fa3c8113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81141d0 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3cddf7700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c810d4c0 con 0x7fa3c8102c70 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3c77fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40734/0 (socket says 172.21.15.55:40734) 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.277+0000 7fa3c77fe700 1 -- 172.21.15.55:0/998474631 learned_addr learned my addr 172.21.15.55:0/998474631 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3c6ffd700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c8102c70 0x7fa3c8113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3c77fe700 1 -- 172.21.15.55:0/998474631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c8102c70 msgr2=0x7fa3c8113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3c77fe700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c8102c70 0x7fa3c8113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.281 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3c77fe700 1 -- 172.21.15.55:0/998474631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8005040 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.282 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3c77fe700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009fd0 tx=0x7fa3b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:15.282 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b800e040 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.282 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c810d6f0 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c810dc30 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3b8013420 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.278+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3b801c630 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.279+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b40052e0 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.280+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa3b800bb80 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.280+0000 7fa3affff700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 0x7fa3b006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.280+0000 7fa3c6ffd700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 0x7fa3b006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.280+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa3b8087c10 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.283 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.281+0000 7fa3c6ffd700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 0x7fa3b006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3c810eea0 tx=0x7fa3bc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:15.287 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.285+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3b8057eb0 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.485+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa3b4005590 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.485+0000 7fa3affff700 1 -- 172.21.15.55:0/998474631 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3b8057a40 con 0x7fa3c80fb1e0 2023-12-19T19:11:15.488 INFO:teuthology.orchestra.run.smithi055.stdout:81604378650 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 msgr2=0x7fa3b006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 0x7fa3b006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa3c810eea0 tx=0x7fa3bc00b040 comp rx=0 tx=0).stop 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 msgr2=0x7fa3c8113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8009fd0 tx=0x7fa3b8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 shutdown_connections 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fa3b0068f70 0x7fa3b006b420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.490 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fa3c80fb1e0 0x7fa3c8113600 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 --2- 172.21.15.55:0/998474631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa3c8102c70 0x7fa3c8113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 >> 172.21.15.55:0/998474631 conn(0x7fa3c80711a0 msgr2=0x7fa3c80fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:15.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 shutdown_connections 2023-12-19T19:11:15.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.488+0000 7fa3cddf7700 1 -- 172.21.15.55:0/998474631 wait complete. 2023-12-19T19:11:15.586 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 -- 172.21.15.55:0/2588930539 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc48101b00 msgr2=0x7efc48101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2588930539 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc48101b00 0x7efc48101ed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7efc40009ce0 tx=0x7efc40004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 -- 172.21.15.55:0/2588930539 shutdown_connections 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2588930539 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc480fbb40 0x7efc480fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2588930539 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc48101b00 0x7efc48101ed0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 -- 172.21.15.55:0/2588930539 >> 172.21.15.55:0/2588930539 conn(0x7efc480719b0 msgr2=0x7efc48071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 -- 172.21.15.55:0/2588930539 shutdown_connections 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.580+0000 7efc4cc02700 1 -- 172.21.15.55:0/2588930539 wait complete. 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 Processor -- start 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 -- start start 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 0x7efc4810ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.587 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4810f260 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4cc02700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4810f3d0 con 0x7efc48101b00 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4659c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:40760/0 (socket says 172.21.15.55:40760) 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4659c700 1 -- 172.21.15.55:0/2975676973 learned_addr learned my addr 172.21.15.55:0/2975676973 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc45d9b700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 0x7efc4810ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4659c700 1 -- 172.21.15.55:0/2975676973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 msgr2=0x7efc4810ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.581+0000 7efc4659c700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 0x7efc4810ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc4659c700 1 -- 172.21.15.55:0/2975676973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40009940 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc45d9b700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 0x7efc4810ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc4659c700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc40009360 tx=0x7efc40004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc4000e040 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc40004610 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc48117700 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc40012b50 con 0x7efc480fbb40 2023-12-19T19:11:15.588 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.582+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc48117bc0 con 0x7efc480fbb40 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.583+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc48103fb0 con 0x7efc480fbb40 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.583+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7efc4001c830 con 0x7efc480fbb40 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.584+0000 7efc3effd700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 0x7efc3406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.584+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc40088c30 con 0x7efc480fbb40 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.584+0000 7efc45d9b700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 0x7efc3406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:15.589 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.584+0000 7efc45d9b700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 0x7efc3406b420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc48065e50 tx=0x7efc3000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:15.590 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.588+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7efc40055440 con 0x7efc480fbb40 2023-12-19T19:11:15.791 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.788+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efc48065b90 con 0x7efc480fbb40 2023-12-19T19:11:15.791 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.789+0000 7efc3effd700 1 -- 172.21.15.55:0/2975676973 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc40058a60 con 0x7efc480fbb40 2023-12-19T19:11:15.792 INFO:teuthology.orchestra.run.smithi055.stdout:60129542176 2023-12-19T19:11:15.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 msgr2=0x7efc3406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 0x7efc3406b420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efc48065e50 tx=0x7efc3000b040 comp rx=0 tx=0).stop 2023-12-19T19:11:15.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 msgr2=0x7efc4810e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:15.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc40009360 tx=0x7efc40004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:15.794 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 shutdown_connections 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7efc34068f70 0x7efc3406b420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7efc480fbb40 0x7efc4810e790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 --2- 172.21.15.55:0/2975676973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc48101b00 0x7efc4810ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 >> 172.21.15.55:0/2975676973 conn(0x7efc480719b0 msgr2=0x7efc480faca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 shutdown_connections 2023-12-19T19:11:15.795 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:15.791+0000 7efc4cc02700 1 -- 172.21.15.55:0/2975676973 wait complete. 2023-12-19T19:11:16.290 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:16.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:15 smithi055 ceph-mon[80978]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:16.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:15 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/998474631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-19T19:11:16.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:15 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2975676973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-19T19:11:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:15 smithi187 ceph-mon[87698]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:15 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/998474631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-19T19:11:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:15 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2975676973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-19T19:11:16.790 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2023-12-19T19:11:16.790 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:17.486 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:17.505 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2023-12-19T19:11:17.506 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:18.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:17 smithi055 ceph-mon[80978]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:18.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:17 smithi187 ceph-mon[87698]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:19.963 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 -- 172.21.15.55:0/2653440397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 msgr2=0x7fc2740fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:19.963 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 --2- 172.21.15.55:0/2653440397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc2740fcc00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc268004940 tx=0x7fc268009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:19.963 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 -- 172.21.15.55:0/2653440397 shutdown_connections 2023-12-19T19:11:19.963 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 --2- 172.21.15.55:0/2653440397 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc2740fcc00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 --2- 172.21.15.55:0/2653440397 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 0x7fc274102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 -- 172.21.15.55:0/2653440397 >> 172.21.15.55:0/2653440397 conn(0x7fc2740f87a0 msgr2=0x7fc2740fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 -- 172.21.15.55:0/2653440397 shutdown_connections 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 -- 172.21.15.55:0/2653440397 wait complete. 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.955+0000 7fc27b75e700 1 Processor -- start 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc27b75e700 1 -- start start 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc27b75e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 0x7fc27406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc27b75e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:19.964 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc27b75e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274070a80 con 0x7fc274103590 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc27b75e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc274070bf0 con 0x7fc274102bf0 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43042/0 (socket says 172.21.15.55:43042) 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 -- 172.21.15.55:0/1694028169 learned_addr learned my addr 172.21.15.55:0/1694028169 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc2794fa700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 0x7fc27406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 -- 172.21.15.55:0/1694028169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 msgr2=0x7fc27406ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 0x7fc27406ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 -- 172.21.15.55:0/1694028169 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268005040 con 0x7fc274103590 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.956+0000 7fc278cf9700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b3b0 tx=0x7fc268009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.957+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26800e040 con 0x7fc274103590 2023-12-19T19:11:19.965 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.957+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc274119860 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.957+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc268013420 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.957+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26801c610 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.957+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc274119cc0 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.958+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2741050c0 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.958+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc268013a20 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.958+0000 7fc271ffb700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 0x7fc25c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.959+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc268087d20 con 0x7fc274103590 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.959+0000 7fc2794fa700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 0x7fc25c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:19.966 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.959+0000 7fc2794fa700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 0x7fc25c06b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc264004fa0 tx=0x7fc26400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:19.967 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:19.963+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc268054530 con 0x7fc274103590 2023-12-19T19:11:20.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:19 smithi055 ceph-mon[80978]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:20.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:20.170 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.165+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc2740fc3a0 con 0x7fc274103590 2023-12-19T19:11:20.170 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.165+0000 7fc271ffb700 1 -- 172.21.15.55:0/1694028169 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc268057b50 con 0x7fc274103590 2023-12-19T19:11:20.171 INFO:teuthology.orchestra.run.smithi055.stdout:150323855376 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 msgr2=0x7fc25c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 0x7fc25c06b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc264004fa0 tx=0x7fc26400b040 comp rx=0 tx=0).stop 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 msgr2=0x7fc274070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b3b0 tx=0x7fc268009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 shutdown_connections 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fc25c068f60 0x7fc25c06b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc274102bf0 0x7fc27406ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.172 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 --2- 172.21.15.55:0/1694028169 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fc274103590 0x7fc274070480 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 >> 172.21.15.55:0/1694028169 conn(0x7fc2740f87a0 msgr2=0x7fc2740fa300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 shutdown_connections 2023-12-19T19:11:20.173 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.168+0000 7fc27b75e700 1 -- 172.21.15.55:0/1694028169 wait complete. 2023-12-19T19:11:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/3674495001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fe590 msgr2=0x7f7e68106c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/3674495001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fe590 0x7f7e68106c60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7e54004940 tx=0x7f7e54009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/3674495001 shutdown_connections 2023-12-19T19:11:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/3674495001 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fe590 0x7f7e68106c60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.188 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/3674495001 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fdbf0 0x7f7e680fdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/3674495001 >> 172.21.15.55:0/3674495001 conn(0x7f7e680f7690 msgr2=0x7f7e680f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/3674495001 shutdown_connections 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.173+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/3674495001 wait complete. 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 Processor -- start 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 -- start start 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:20.189 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fe590 0x7f7e68113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68114100 con 0x7f7e680fdbf0 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6ed8c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6810d430 con 0x7f7e680fe590 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6cb28700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6cb28700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43068/0 (socket says 172.21.15.55:43068) 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e6cb28700 1 -- 172.21.15.55:0/1290663464 learned_addr learned my addr 172.21.15.55:0/1290663464 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.174+0000 7f7e67fff700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fe590 0x7f7e68113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:20.190 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6cb28700 1 -- 172.21.15.55:0/1290663464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fe590 msgr2=0x7f7e68113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6cb28700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fe590 0x7f7e68113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6cb28700 1 -- 172.21.15.55:0/1290663464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e54005040 con 0x7f7e680fdbf0 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6cb28700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c002a80 tx=0x7f7e5c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5c007cc0 con 0x7f7e680fdbf0 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e5c00f040 con 0x7f7e680fdbf0 2023-12-19T19:11:20.191 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e6810d710 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e5c00da60 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.175+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6810dc20 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.176+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e681042c0 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.176+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7e5c00dbc0 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.177+0000 7f7e657fa700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 0x7f7e5806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.177+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e5c086610 con 0x7f7e680fdbf0 2023-12-19T19:11:20.192 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.177+0000 7f7e67fff700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 0x7f7e5806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:20.193 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.177+0000 7f7e67fff700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 0x7f7e5806b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e6810e760 tx=0x7f7e54009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:20.193 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.181+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7e5c0568b0 con 0x7f7e680fdbf0 2023-12-19T19:11:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:19 smithi187 ceph-mon[87698]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:20.383 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.380+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7e680647b0 con 0x7f7e680fdbf0 2023-12-19T19:11:20.383 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.380+0000 7f7e657fa700 1 -- 172.21.15.55:0/1290663464 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7e5c056440 con 0x7f7e680fdbf0 2023-12-19T19:11:20.383 INFO:teuthology.orchestra.run.smithi055.stdout:38654705702 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 msgr2=0x7f7e5806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 0x7f7e5806b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7e6810e760 tx=0x7f7e54009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 msgr2=0x7f7e68113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c002a80 tx=0x7f7e5c0046b0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 shutdown_connections 2023-12-19T19:11:20.386 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7e58068f60 0x7f7e5806b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.387 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7e680fdbf0 0x7f7e68113570 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.387 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 --2- 172.21.15.55:0/1290663464 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7e680fe590 0x7f7e68113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:20.387 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 >> 172.21.15.55:0/1290663464 conn(0x7f7e680f7690 msgr2=0x7f7e68101440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:20.387 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 shutdown_connections 2023-12-19T19:11:20.387 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:20.384+0000 7f7e6ed8c700 1 -- 172.21.15.55:0/1290663464 wait complete. 2023-12-19T19:11:20.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855376 for osd.4 2023-12-19T19:11:20.879 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:21.247 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:20 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1694028169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-19T19:11:21.247 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:20 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1290663464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-19T19:11:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:20 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1694028169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-19T19:11:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:20 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1290663464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-19T19:11:21.964 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2023-12-19T19:11:21.965 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:22.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:21 smithi055 ceph-mon[80978]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:21 smithi187 ceph-mon[87698]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- 172.21.15.55:0/2760261719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f80102c00 msgr2=0x7f3f80102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2760261719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f80102c00 0x7f3f80102fd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c004940 tx=0x7f3f7c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- 172.21.15.55:0/2760261719 shutdown_connections 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2760261719 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f801035a0 0x7f3f800fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2760261719 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f80102c00 0x7f3f80102fd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- 172.21.15.55:0/2760261719 >> 172.21.15.55:0/2760261719 conn(0x7f3f800f8790 msgr2=0x7f3f800faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- 172.21.15.55:0/2760261719 shutdown_connections 2023-12-19T19:11:22.435 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- 172.21.15.55:0/2760261719 wait complete. 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 Processor -- start 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.432+0000 7f3f8780c700 1 -- start start 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f8780c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f80102c00 0x7f3f80113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f8780c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f8780c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8010d510 con 0x7f3f801035a0 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f8780c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8010d680 con 0x7f3f80102c00 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43086/0 (socket says 172.21.15.55:43086) 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 -- 172.21.15.55:0/2086405028 learned_addr learned my addr 172.21.15.55:0/2086405028 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f855a8700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f80102c00 0x7f3f80113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 -- 172.21.15.55:0/2086405028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f80102c00 msgr2=0x7f3f80113650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f80102c00 0x7f3f80113650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 -- 172.21.15.55:0/2086405028 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c005040 con 0x7f3f801035a0 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f84da7700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000b790 tx=0x7f3f70009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:22.436 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f7000f6f0 con 0x7f3f801035a0 2023-12-19T19:11:22.437 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8010d960 con 0x7f3f801035a0 2023-12-19T19:11:22.437 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f7000fd00 con 0x7f3f801035a0 2023-12-19T19:11:22.437 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.433+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f700103a0 con 0x7f3f801035a0 2023-12-19T19:11:22.437 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.434+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8010dea0 con 0x7f3f801035a0 2023-12-19T19:11:22.437 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.435+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f800fc390 con 0x7f3f801035a0 2023-12-19T19:11:22.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.435+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3f7000f850 con 0x7f3f801035a0 2023-12-19T19:11:22.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.436+0000 7f3f75ffb700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 0x7f3f6c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:22.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.436+0000 7f3f855a8700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 0x7f3f6c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:22.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.436+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3f70086810 con 0x7f3f801035a0 2023-12-19T19:11:22.438 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.436+0000 7f3f855a8700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 0x7f3f6c06b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00b840 tx=0x7f3f7c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:22.441 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.440+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3f70056ab0 con 0x7f3f801035a0 2023-12-19T19:11:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.635+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3f8010ec30 con 0x7f3f801035a0 2023-12-19T19:11:22.637 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.636+0000 7f3f75ffb700 1 -- 172.21.15.55:0/2086405028 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3f70056640 con 0x7f3f801035a0 2023-12-19T19:11:22.637 INFO:teuthology.orchestra.run.smithi055.stdout:171798691851 2023-12-19T19:11:22.639 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 msgr2=0x7f3f6c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:22.639 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 0x7f3f6c06b420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00b840 tx=0x7f3f7c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 msgr2=0x7f3f80113bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000b790 tx=0x7f3f70009220 comp rx=0 tx=0).stop 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 shutdown_connections 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f3f6c068f70 0x7f3f6c06b420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f80102c00 0x7f3f80113650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 --2- 172.21.15.55:0/2086405028 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f3f801035a0 0x7f3f80113bd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 >> 172.21.15.55:0/2086405028 conn(0x7f3f800f8790 msgr2=0x7f3f800fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 shutdown_connections 2023-12-19T19:11:22.640 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:22.638+0000 7f3f8780c700 1 -- 172.21.15.55:0/2086405028 wait complete. 2023-12-19T19:11:23.295 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691851 for osd.5 2023-12-19T19:11:23.295 DEBUG:teuthology.parallel:result is None 2023-12-19T19:11:23.295 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-19T19:11:23.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-19T19:11:23.566 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:23.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:23 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2086405028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-19T19:11:23.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:23 smithi187 ceph-mon[87698]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:23.903 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:23 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2086405028' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-19T19:11:23.903 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:23 smithi055 ceph-mon[80978]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 -- 172.21.15.55:0/980055767 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0065390 msgr2=0x7f39f0065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 --2- 172.21.15.55:0/980055767 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0065390 0x7f39f0065800 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f39e0004940 tx=0x7f39e0009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 -- 172.21.15.55:0/980055767 shutdown_connections 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 --2- 172.21.15.55:0/980055767 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0065390 0x7f39f0065800 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 --2- 172.21.15.55:0/980055767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0102c20 0x7f39f0102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 -- 172.21.15.55:0/980055767 >> 172.21.15.55:0/980055767 conn(0x7f39f00719b0 msgr2=0x7f39f0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 -- 172.21.15.55:0/980055767 shutdown_connections 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 -- 172.21.15.55:0/980055767 wait complete. 2023-12-19T19:11:25.252 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.248+0000 7f39f82d0700 1 Processor -- start 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f82d0700 1 -- start start 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f82d0700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0065390 0x7f39f0113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f82d0700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f82d0700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f01142c0 con 0x7f39f0102c20 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f82d0700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f010d5f0 con 0x7f39f0065390 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:43102/0 (socket says 172.21.15.55:43102) 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 -- 172.21.15.55:0/3849767130 learned_addr learned my addr 172.21.15.55:0/3849767130 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 -- 172.21.15.55:0/3849767130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0065390 msgr2=0x7f39f0113730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f606c700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0065390 0x7f39f0113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0065390 0x7f39f0113730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 -- 172.21.15.55:0/3849767130 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0005040 con 0x7f39f0102c20 2023-12-19T19:11:25.253 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39f586b700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b3b0 tx=0x7f39e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:25.254 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.249+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e000e040 con 0x7f39f0102c20 2023-12-19T19:11:25.254 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.250+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f010d870 con 0x7f39f0102c20 2023-12-19T19:11:25.254 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.250+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39e000bc50 con 0x7f39f0102c20 2023-12-19T19:11:25.254 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.250+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39e0013420 con 0x7f39f0102c20 2023-12-19T19:11:25.254 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.250+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f010ddb0 con 0x7f39f0102c20 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.251+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f00fff40 con 0x7f39f0102c20 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.251+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f39e001c430 con 0x7f39f0102c20 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.251+0000 7f39e6ffd700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 0x7f39dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.252+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f39e0087850 con 0x7f39f0102c20 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.252+0000 7f39f606c700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 0x7f39dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:25.255 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.252+0000 7f39f606c700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 0x7f39dc06b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0041d0 tx=0x7f39ec009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:25.258 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.256+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f39e0057ba0 con 0x7f39f0102c20 2023-12-19T19:11:25.447 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.445+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39f004bb80 con 0x7f39dc068f60 2023-12-19T19:11:25.449 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.446+0000 7f39e6ffd700 1 -- 172.21.15.55:0/3849767130 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28612 (secure 0 0 0) 0x7f39f004bb80 con 0x7f39dc068f60 2023-12-19T19:11:25.449 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.448+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 msgr2=0x7f39dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.448+0000 7f39f82d0700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 0x7f39dc06b410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f39ec0041d0 tx=0x7f39ec009040 comp rx=0 tx=0).stop 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.448+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 msgr2=0x7f39f0113c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.448+0000 7f39f82d0700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f39e000b3b0 tx=0x7f39e0009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 shutdown_connections 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f39dc068f60 0x7f39dc06b410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f39f0065390 0x7f39f0113730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 --2- 172.21.15.55:0/3849767130 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f39f0102c20 0x7f39f0113c70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 >> 172.21.15.55:0/3849767130 conn(0x7f39f00719b0 msgr2=0x7f39f00fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 shutdown_connections 2023-12-19T19:11:25.451 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:25.449+0000 7f39f82d0700 1 -- 172.21.15.55:0/3849767130 wait complete. 2023-12-19T19:11:25.452 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-19T19:11:26.192 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:25 smithi055 ceph-mon[80978]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:26.222 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2023-12-19T19:11:24.886470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787648,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683264,"statfs":{"total":575970213888,"available":574139662336,"internally_reserved":0,"allocated":4014080,"data_stored":2624143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002172"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-12-19T19:10:32.597657+0000","last_change":"2023-12-19T19:10:06.622821+0000","last_active":"2023-12-19T19:10:32.597657+0000","last_peered":"2023-12-19T19:10:32.597657+0000","last_clean":"2023-12-19T19:10:32.597657+0000","last_became_active":"2023-12-19T19:10:06.622549+0000","last_became_peered":"2023-12-19T19:10:06.622549+0000","last_unstale":"2023-12-19T19:10:32.597657+0000","last_undegraded":"2023-12-19T19:10:32.597657+0000","last_fullsized":"2023-12-19T19:10:32.597657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T19:09:19.656293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T19:09:19.656293+0000","last_clean_scrub_stamp":"2023-12-19T19:09:19.656293+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:50:51.003777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42799999999999999}]},{"osd":1,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.44}]},{"osd":3,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298228,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446924,"statfs":{"total":95995035648,"available":95689650176,"internally_reserved":0,"allocated":962560,"data_stored":732272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55400000000000005}]},{"osd":1,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54000000000000004}]},{"osd":4,"last update":"Tue Dec 19 19:11:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.434}]},{"osd":3,"last update":"Tue Dec 19 19:10:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.82699999999999996}]},{"osd":4,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":16.07,"5min":16.07,"15min":16.07},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 19:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.504,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.25,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.318}]},{"osd":2,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.34100000000000003}]},{"osd":3,"last update":"Tue Dec 19 19:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.63200000000000001}]},{"osd":4,"last update":"Tue Dec 19 19:11:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.33500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55500000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.34300000000000003},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Dec 19 19:10:23 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.44900000000000001}]},{"osd":3,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.76200000000000001}]},{"osd":4,"last update":"Tue Dec 19 19:11:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.79900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T19:11:26.223 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-19T19:11:26.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:25 smithi187 ceph-mon[87698]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:26.498 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:27.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:26 smithi055 ceph-mon[80978]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:11:27.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:26 smithi187 ceph-mon[87698]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:11:28.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:27 smithi055 ceph-mon[80978]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:28.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 -- 172.21.15.55:0/331474770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 msgr2=0x7fe53c0fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:28.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 --2- 172.21.15.55:0/331474770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fb630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe52c004940 tx=0x7fe52c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:28.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 -- 172.21.15.55:0/331474770 shutdown_connections 2023-12-19T19:11:28.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 --2- 172.21.15.55:0/331474770 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fb630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.110 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 --2- 172.21.15.55:0/331474770 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 -- 172.21.15.55:0/331474770 >> 172.21.15.55:0/331474770 conn(0x7fe53c071140 msgr2=0x7fe53c071540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 -- 172.21.15.55:0/331474770 shutdown_connections 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 -- 172.21.15.55:0/331474770 wait complete. 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.106+0000 7fe5438ab700 1 Processor -- start 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe5438ab700 1 -- start start 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe5438ab700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe5438ab700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c0fc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe5438ab700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0fcee0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe5438ab700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c0fd050 con 0x7fe53c102c10 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52372/0 (socket says 172.21.15.55:52372) 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 -- 172.21.15.55:0/3928437607 learned_addr learned my addr 172.21.15.55:0/3928437607 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 -- 172.21.15.55:0/3928437607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 msgr2=0x7fe53c0fc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe540e46700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c0fc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:28.111 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c0fc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 -- 172.21.15.55:0/3928437607 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52c005040 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe540e46700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c0fc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe541647700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe538002a50 tx=0x7fe5380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.107+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe538007cc0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.108+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe53800f040 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.108+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c1198c0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.108+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe538009540 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.112 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.108+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c119da0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.109+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c0ffef0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.109+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe5380096a0 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.109+0000 7fe531ffb700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 0x7fe52806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.109+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe538086890 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.110+0000 7fe540e46700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 0x7fe52806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:28.113 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.110+0000 7fe540e46700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 0x7fe52806b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009fd0 tx=0x7fe52c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:28.116 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.114+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe538053020 con 0x7fe53c0fb1c0 2023-12-19T19:11:28.308 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.305+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe53c0fdae0 con 0x7fe528068f60 2023-12-19T19:11:28.309 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.307+0000 7fe531ffb700 1 -- 172.21.15.55:0/3928437607 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28613 (secure 0 0 0) 0x7fe53c0fdae0 con 0x7fe528068f60 2023-12-19T19:11:28.310 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:28.312 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 msgr2=0x7fe52806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 0x7fe52806b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009fd0 tx=0x7fe52c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 msgr2=0x7fe53c0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe538002a50 tx=0x7fe5380046b0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 shutdown_connections 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe528068f60 0x7fe52806b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe53c0fb1c0 0x7fe53c0fc3a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 --2- 172.21.15.55:0/3928437607 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe53c102c10 0x7fe53c0fc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 >> 172.21.15.55:0/3928437607 conn(0x7fe53c071140 msgr2=0x7fe53c0fa340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 shutdown_connections 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:28.309+0000 7fe5438ab700 1 -- 172.21.15.55:0/3928437607 wait complete. 2023-12-19T19:11:28.313 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-19T19:11:28.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:27 smithi187 ceph-mon[87698]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:28.919 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2023-12-19T19:11:26.886822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787648,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683264,"statfs":{"total":575970213888,"available":574139662336,"internally_reserved":0,"allocated":4014080,"data_stored":2624143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002210"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2023-12-19T19:10:32.597657+0000","last_change":"2023-12-19T19:10:06.622821+0000","last_active":"2023-12-19T19:10:32.597657+0000","last_peered":"2023-12-19T19:10:32.597657+0000","last_clean":"2023-12-19T19:10:32.597657+0000","last_became_active":"2023-12-19T19:10:06.622549+0000","last_became_peered":"2023-12-19T19:10:06.622549+0000","last_unstale":"2023-12-19T19:10:32.597657+0000","last_undegraded":"2023-12-19T19:10:32.597657+0000","last_fullsized":"2023-12-19T19:10:32.597657+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T19:09:19.656293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T19:09:19.656293+0000","last_clean_scrub_stamp":"2023-12-19T19:09:19.656293+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-21T00:50:51.003777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53200000000000003}]},{"osd":3,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.499}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298228,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446924,"statfs":{"total":95995035648,"available":95689650176,"internally_reserved":0,"allocated":962560,"data_stored":732272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.55400000000000005}]},{"osd":1,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.54000000000000004}]},{"osd":4,"last update":"Tue Dec 19 19:11:07 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.442}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.434}]},{"osd":3,"last update":"Tue Dec 19 19:10:43 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.82699999999999996}]},{"osd":4,"last update":"Tue Dec 19 19:11:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":16.07,"5min":16.07,"15min":16.07},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 19 19:10:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.33400000000000002,"15min":0.33200000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.504,"5min":0.50800000000000001,"15min":0.50800000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.34100000000000003,"15min":0.33700000000000002},"min":{"1min":0.25,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.318}]},{"osd":2,"last update":"Tue Dec 19 19:10:21 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.34100000000000003}]},{"osd":3,"last update":"Tue Dec 19 19:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.63200000000000001}]},{"osd":4,"last update":"Tue Dec 19 19:11:10 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 19 19:10:52 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.33500000000000002},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.55500000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.35199999999999998,"15min":0.34300000000000003},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.28699999999999998}]},{"osd":2,"last update":"Tue Dec 19 19:10:23 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.30299999999999999}]},{"osd":3,"last update":"Tue Dec 19 19:10:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52400000000000002}]},{"osd":4,"last update":"Tue Dec 19 19:11:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-19T19:11:28.920 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-19T19:11:28.920 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-19T19:11:28.920 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-19T19:11:28.920 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-19T19:11:29.191 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:29 smithi055 ceph-mon[80978]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:11:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:29 smithi055 ceph-mon[80978]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:30.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:29 smithi187 ceph-mon[87698]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-19T19:11:30.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:29 smithi187 ceph-mon[87698]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3573075601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f780fbb00 msgr2=0x7f1f780fbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3573075601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f780fbb00 0x7f1f780fbf70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c004940 tx=0x7f1f6c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3573075601 shutdown_connections 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3573075601 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f780fbb00 0x7f1f780fbf70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3573075601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f78101b60 0x7f1f78101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3573075601 >> 172.21.15.55:0/3573075601 conn(0x7f1f780f7680 msgr2=0x7f1f780f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3573075601 shutdown_connections 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.864+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3573075601 wait complete. 2023-12-19T19:11:30.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 Processor -- start 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 -- start start 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f780fbb00 0x7f1f7810e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7810f240 con 0x7f1f78101b60 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f7dfb8700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7810f3b0 con 0x7f1f780fbb00 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52380/0 (socket says 172.21.15.55:52380) 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 -- 172.21.15.55:0/3122443425 learned_addr learned my addr 172.21.15.55:0/3122443425 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f777fe700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f780fbb00 0x7f1f7810e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 -- 172.21.15.55:0/3122443425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f780fbb00 msgr2=0x7f1f7810e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f780fbb00 0x7f1f7810e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 -- 172.21.15.55:0/3122443425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c005040 con 0x7f1f78101b60 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.865+0000 7f1f76ffd700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009380 tx=0x7f1f6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:30.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.866+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f6c00e040 con 0x7f1f78101b60 2023-12-19T19:11:30.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.866+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c00bc50 con 0x7f1f78101b60 2023-12-19T19:11:30.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.866+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f78117620 con 0x7f1f78101b60 2023-12-19T19:11:30.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.866+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f6c013660 con 0x7f1f78101b60 2023-12-19T19:11:30.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.866+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f78117b60 con 0x7f1f78101b60 2023-12-19T19:11:30.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.867+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f78103f40 con 0x7f1f78101b60 2023-12-19T19:11:30.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.867+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1f6c01c430 con 0x7f1f78101b60 2023-12-19T19:11:30.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.867+0000 7f1f5ffff700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 0x7f1f6006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:30.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.868+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1f6c0876a0 con 0x7f1f78101b60 2023-12-19T19:11:30.872 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.868+0000 7f1f777fe700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 0x7f1f6006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:30.872 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.868+0000 7f1f777fe700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 0x7f1f6006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f68004d60 tx=0x7f1f6800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:30.875 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:30.873+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1f6c054000 con 0x7f1f78101b60 2023-12-19T19:11:31.115 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.112+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1f7804bbb0 con 0x7f1f78101b60 2023-12-19T19:11:31.115 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.112+0000 7f1f5ffff700 1 -- 172.21.15.55:0/3122443425 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1f6c057620 con 0x7f1f78101b60 2023-12-19T19:11:31.115 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:31.115 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 msgr2=0x7f1f6006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 0x7f1f6006b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1f68004d60 tx=0x7f1f6800b040 comp rx=0 tx=0).stop 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 msgr2=0x7f1f7810ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c009380 tx=0x7f1f6c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 shutdown_connections 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1f60068f60 0x7f1f6006b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:31.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1f780fbb00 0x7f1f7810e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:31.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 --2- 172.21.15.55:0/3122443425 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f1f78101b60 0x7f1f7810ebf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:31.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 >> 172.21.15.55:0/3122443425 conn(0x7f1f780f7680 msgr2=0x7f1f780f8350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:31.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 shutdown_connections 2023-12-19T19:11:31.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:31.115+0000 7f1f7dfb8700 1 -- 172.21.15.55:0/3122443425 wait complete. 2023-12-19T19:11:31.835 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-12-19T19:11:31.835 INFO:tasks.cephadm:Setup complete, yielding 2023-12-19T19:11:31.835 INFO:teuthology.run_tasks:Running task print... 2023-12-19T19:11:31.845 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2023-12-19T19:11:31.845 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:11:31.853 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:11:31.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2023-12-19T19:11:32.123 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:32.244 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:31 smithi055 ceph-mon[80978]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:32.244 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:31 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3122443425' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-19T19:11:32.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:31 smithi187 ceph-mon[87698]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:32.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:31 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3122443425' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-19T19:11:33.831 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 -- 172.21.15.55:0/1194948303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 msgr2=0x7f11380fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:33.831 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1194948303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f11380fbf00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1128004940 tx=0x7f1128009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 -- 172.21.15.55:0/1194948303 shutdown_connections 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1194948303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 0x7f11380fd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1194948303 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f11380fbf00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 -- 172.21.15.55:0/1194948303 >> 172.21.15.55:0/1194948303 conn(0x7f11380f7270 msgr2=0x7f11380f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 -- 172.21.15.55:0/1194948303 shutdown_connections 2023-12-19T19:11:33.832 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.828+0000 7f113d1bf700 1 -- 172.21.15.55:0/1194948303 wait complete. 2023-12-19T19:11:33.832 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:33 smithi055 ceph-mon[80978]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.829+0000 7f113d1bf700 1 Processor -- start 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.829+0000 7f113d1bf700 1 -- start start 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113d1bf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113d1bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 0x7f113810dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113d1bf700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113810e0e0 con 0x7f11380fbaf0 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113d1bf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113810e980 con 0x7f11380fcda0 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52392/0 (socket says 172.21.15.55:52392) 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 -- 172.21.15.55:0/1442795495 learned_addr learned my addr 172.21.15.55:0/1442795495 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113659c700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 0x7f113810dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 -- 172.21.15.55:0/1442795495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 msgr2=0x7f113810dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:33.834 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 0x7f113810dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 -- 172.21.15.55:0/1442795495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1128005040 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f1136d9d700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f112800b3b0 tx=0x7f1128009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f112800e040 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113810ec00 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.830+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1128013420 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.831+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f112801c650 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.831+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113810f140 con 0x7f11380fbaf0 2023-12-19T19:11:33.835 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.831+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113804bb80 con 0x7f11380fbaf0 2023-12-19T19:11:33.836 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.832+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1128013590 con 0x7f11380fbaf0 2023-12-19T19:11:33.836 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.833+0000 7f112f7fe700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 0x7f112406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:33.836 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.833+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1128087c90 con 0x7f11380fbaf0 2023-12-19T19:11:33.836 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.833+0000 7f113659c700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 0x7f112406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:33.836 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.833+0000 7f113659c700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 0x7f112406b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1120004f40 tx=0x7f112000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:33.839 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:33.837+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1128058080 con 0x7f11380fbaf0 2023-12-19T19:11:34.033 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.031+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f113804bb80 con 0x7f11380fbaf0 2023-12-19T19:11:34.038 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.035+0000 7f112f7fe700 1 -- 172.21.15.55:0/1442795495 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f1128057c10 con 0x7f11380fbaf0 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 msgr2=0x7f112406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 0x7f112406b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1120004f40 tx=0x7f112000b040 comp rx=0 tx=0).stop 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 msgr2=0x7f113810d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f112800b3b0 tx=0x7f1128009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 shutdown_connections 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f1124068f60 0x7f112406b410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f11380fbaf0 0x7f113810d5a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:34.040 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 --2- 172.21.15.55:0/1442795495 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f11380fcda0 0x7f113810dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:34.041 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 >> 172.21.15.55:0/1442795495 conn(0x7f11380f7270 msgr2=0x7f1138065c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:34.041 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 shutdown_connections 2023-12-19T19:11:34.041 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:34.038+0000 7f113d1bf700 1 -- 172.21.15.55:0/1442795495 wait complete. 2023-12-19T19:11:34.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:33 smithi187 ceph-mon[87698]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:34.803 INFO:teuthology.run_tasks:Running task print... 2023-12-19T19:11:34.812 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2023-12-19T19:11:34.812 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:11:34.821 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:11:34.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph orch status' 2023-12-19T19:11:35.098 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1442795495' entity='client.admin' 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:35 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1442795495' entity='client.admin' 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:35.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:35 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:36.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 -- 172.21.15.55:0/2252546375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 msgr2=0x7f87280fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:36.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 --2- 172.21.15.55:0/2252546375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f87280fe3b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f871c004940 tx=0x7f871c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:36.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 -- 172.21.15.55:0/2252546375 shutdown_connections 2023-12-19T19:11:36.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 --2- 172.21.15.55:0/2252546375 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f87280fe3b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.703 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 --2- 172.21.15.55:0/2252546375 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 0x7f87280fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 -- 172.21.15.55:0/2252546375 >> 172.21.15.55:0/2252546375 conn(0x7f87280f6950 msgr2=0x7f87280f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 -- 172.21.15.55:0/2252546375 shutdown_connections 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.699+0000 7f8730185700 1 -- 172.21.15.55:0/2252546375 wait complete. 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 Processor -- start 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 -- start start 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 0x7f872810d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810e190 con 0x7f87280fdf60 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f8730185700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810ea30 con 0x7f87280faed0 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f872d720700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f872d720700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:52408/0 (socket says 172.21.15.55:52408) 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f872d720700 1 -- 172.21.15.55:0/2377370296 learned_addr learned my addr 172.21.15.55:0/2377370296 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.700+0000 7f872df21700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 0x7f872810d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f872d720700 1 -- 172.21.15.55:0/2377370296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 msgr2=0x7f872810d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:36.704 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f872d720700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 0x7f872810d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f872d720700 1 -- 172.21.15.55:0/2377370296 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c005040 con 0x7f87280fdf60 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f872d720700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b3b0 tx=0x7f871c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871c00e040 con 0x7f87280fdf60 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872810ecb0 con 0x7f87280fdf60 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f871c013420 con 0x7f87280fdf60 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f871c01c650 con 0x7f87280fdf60 2023-12-19T19:11:36.705 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.701+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872810f1f0 con 0x7f87280fdf60 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.702+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87281075f0 con 0x7f87280fdf60 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.702+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f871c013590 con 0x7f87280fdf60 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.703+0000 7f871a7fc700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 0x7f871406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.703+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f871c087eb0 con 0x7f87280fdf60 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.703+0000 7f872df21700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 0x7f871406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:36.706 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.703+0000 7f872df21700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 0x7f871406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8728065960 tx=0x7f8724009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:36.710 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.707+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f871c058160 con 0x7f87280fdf60 2023-12-19T19:11:36.922 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.920+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f872810f4d0 con 0x7f8714068f60 2023-12-19T19:11:36.924 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.921+0000 7f871a7fc700 1 -- 172.21.15.55:0/2377370296 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f872810f4d0 con 0x7f8714068f60 2023-12-19T19:11:36.924 INFO:teuthology.orchestra.run.smithi055.stdout:Backend: cephadm 2023-12-19T19:11:36.924 INFO:teuthology.orchestra.run.smithi055.stdout:Available: Yes 2023-12-19T19:11:36.924 INFO:teuthology.orchestra.run.smithi055.stdout:Paused: No 2023-12-19T19:11:36.926 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.923+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 msgr2=0x7f871406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:36.926 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.923+0000 7f8730185700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 0x7f871406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8728065960 tx=0x7f8724009040 comp rx=0 tx=0).stop 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.923+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 msgr2=0x7f872810db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.923+0000 7f8730185700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b3b0 tx=0x7f871c009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 shutdown_connections 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f8714068f60 0x7f871406b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f87280faed0 0x7f872810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 --2- 172.21.15.55:0/2377370296 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f87280fdf60 0x7f872810db90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:36.927 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 >> 172.21.15.55:0/2377370296 conn(0x7f87280f6950 msgr2=0x7f87280f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:36.928 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 shutdown_connections 2023-12-19T19:11:36.928 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:36.924+0000 7f8730185700 1 -- 172.21.15.55:0/2377370296 wait complete. 2023-12-19T19:11:37.520 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ps' 2023-12-19T19:11:37.790 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:38.215 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:37 smithi055 ceph-mon[80978]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:38.216 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:37 smithi055 ceph-mon[80978]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:38.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:37 smithi187 ceph-mon[87698]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:38.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:37 smithi187 ceph-mon[87698]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 -- 172.21.15.55:0/1434575070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 msgr2=0x7f09740fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 --2- 172.21.15.55:0/1434575070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f09740fb6a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0964004940 tx=0x7f0964009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 -- 172.21.15.55:0/1434575070 shutdown_connections 2023-12-19T19:11:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 --2- 172.21.15.55:0/1434575070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f09740fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 --2- 172.21.15.55:0/1434575070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f09740fb6a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.346 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 -- 172.21.15.55:0/1434575070 >> 172.21.15.55:0/1434575070 conn(0x7f09740f6950 msgr2=0x7f09740f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:39.346 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 -- 172.21.15.55:0/1434575070 shutdown_connections 2023-12-19T19:11:39.346 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.342+0000 7f097a806700 1 -- 172.21.15.55:0/1434575070 wait complete. 2023-12-19T19:11:39.346 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 Processor -- start 2023-12-19T19:11:39.346 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 -- start start 2023-12-19T19:11:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f097410ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097410d020 con 0x7f09740fb290 2023-12-19T19:11:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f097a806700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097410d8c0 con 0x7f09740fc4a0 2023-12-19T19:11:39.347 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f0973fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f0973fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33076/0 (socket says 172.21.15.55:33076) 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.343+0000 7f0973fff700 1 -- 172.21.15.55:0/3631556114 learned_addr learned my addr 172.21.15.55:0/3631556114 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f09737fe700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f097410ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0973fff700 1 -- 172.21.15.55:0/3631556114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 msgr2=0x7f097410ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0973fff700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f097410ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0973fff700 1 -- 172.21.15.55:0/3631556114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0964005040 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f09737fe700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f097410ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0973fff700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0964009390 tx=0x7f0964009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f096400e040 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097410db40 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0964004290 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0964013660 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.344+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097410e020 con 0x7f09740fb290 2023-12-19T19:11:39.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.345+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097404bb80 con 0x7f09740fb290 2023-12-19T19:11:39.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.346+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f096401ea40 con 0x7f09740fb290 2023-12-19T19:11:39.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.346+0000 7f0970ff9700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 0x7f095c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:39.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.346+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f096408a9f0 con 0x7f09740fb290 2023-12-19T19:11:39.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.346+0000 7f09737fe700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 0x7f095c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:39.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.347+0000 7f09737fe700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 0x7f095c06b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09680041d0 tx=0x7f0968009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:39.353 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.351+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f096408ce60 con 0x7f09740fb290 2023-12-19T19:11:39.557 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.555+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f097410e470 con 0x7f095c068f70 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.564+0000 7f0970ff9700 1 -- 172.21.15.55:0/3631556114 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f097410e470 con 0x7f095c068f70 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager.smithi055 smithi055 *:9093,9094 running (3m) 2m ago 5m 25.8M - 0.25.0 c8568f914cd2 526d8c1dbdb9 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi055 smithi055 running (5m) 2m ago 5m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 53b90cb7fb26 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi187 smithi187 running (4m) 69s ago 4m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 52d1b402aa76 2023-12-19T19:11:39.566 INFO:teuthology.orchestra.run.smithi055.stdout:grafana.smithi055 smithi055 *:3000 running (3m) 2m ago 4m 77.8M - 9.4.7 954c08fa6188 c4acfbe0e286 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi055.dwclvp smithi055 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-131-ga88f3edf f9ecdbaa1030 c99d4ccfe879 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi187.pmhvpw smithi187 *:8443,9283 running (4m) 69s ago 4m 432M - 17.2.7-131-ga88f3edf f9ecdbaa1030 36e60396eb8d 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi055 smithi055 running (6m) 2m ago 6m 43.0M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 22d858bf8f44 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi187 smithi187 running (4m) 69s ago 4m 41.2M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 31c58c5214bc 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi055 smithi055 *:9100 running (5m) 2m ago 5m 15.6M - 1.5.0 0da6a335fe13 4477ddc2ccc2 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi187 smithi187 *:9100 running (4m) 69s ago 4m 17.3M - 1.5.0 0da6a335fe13 8a84ed919750 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 smithi055 running (3m) 2m ago 3m 38.8M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 b82fbcd42bf0 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 smithi055 running (2m) 2m ago 2m 41.7M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0e674f452739 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 smithi055 running (2m) 2m ago 2m 40.6M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 338fa21e9e0b 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 smithi187 running (2m) 69s ago 2m 38.3M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 153484b5fa97 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 smithi187 running (99s) 69s ago 98s 36.7M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 96c21caf7e77 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 smithi187 running (74s) 69s ago 73s 10.0M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 7c6c250a7b10 2023-12-19T19:11:39.567 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus.smithi055 smithi055 *:9095 running (3m) 2m ago 4m 37.9M - 2.43.0 a07b618ecd1d a51dcacc4149 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 msgr2=0x7f095c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 0x7f095c06b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09680041d0 tx=0x7f0968009040 comp rx=0 tx=0).stop 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 msgr2=0x7f097410c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0964009390 tx=0x7f0964009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 shutdown_connections 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f095c068f70 0x7f095c06b420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.568 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f09740fb290 0x7f097410c4e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.569 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.565+0000 7f097a806700 1 --2- 172.21.15.55:0/3631556114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f09740fc4a0 0x7f097410ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:39.569 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.566+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 >> 172.21.15.55:0/3631556114 conn(0x7f09740f6950 msgr2=0x7f09740f8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:39.569 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.566+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 shutdown_connections 2023-12-19T19:11:39.569 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:39.566+0000 7f097a806700 1 -- 172.21.15.55:0/3631556114 wait complete. 2023-12-19T19:11:40.229 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:39 smithi055 ceph-mon[80978]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:40.258 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ls' 2023-12-19T19:11:40.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:39 smithi187 ceph-mon[87698]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:40.528 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:41.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:40 smithi187 ceph-mon[87698]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:41.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:40 smithi055 ceph-mon[80978]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:42.098 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- 172.21.15.55:0/1022356977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fde00 msgr2=0x7fe5fc0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:42.098 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 --2- 172.21.15.55:0/1022356977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc0fe250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec004940 tx=0x7fe5ec009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:42.098 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- 172.21.15.55:0/1022356977 shutdown_connections 2023-12-19T19:11:42.098 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 --2- 172.21.15.55:0/1022356977 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc0fe250 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 --2- 172.21.15.55:0/1022356977 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- 172.21.15.55:0/1022356977 >> 172.21.15.55:0/1022356977 conn(0x7fe5fc0f82d0 msgr2=0x7fe5fc0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- 172.21.15.55:0/1022356977 shutdown_connections 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- 172.21.15.55:0/1022356977 wait complete. 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 Processor -- start 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.094+0000 7fe60468f700 1 -- start start 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:42.099 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10dfd0 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10e870 con 0x7fe5fc0fde00 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33080/0 (socket says 172.21.15.55:33080) 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 -- 172.21.15.55:0/486187214 learned_addr learned my addr 172.21.15.55:0/486187214 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe601c2a700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 -- 172.21.15.55:0/486187214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fde00 msgr2=0x7fe5fc10d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc10d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 -- 172.21.15.55:0/486187214 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec005040 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60242b700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800eb70 tx=0x7fe5f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f8007cc0 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.100 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5f800f040 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.101 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f800c9d0 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.101 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc10eaf0 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.101 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.095+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc10f030 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.096+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc04bb80 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.097+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe5f800cc40 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.097+0000 7fe5f2ffd700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 0x7fe5e806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.097+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5f8086510 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.097+0000 7fe601c2a700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 0x7fe5e806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:42.102 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.098+0000 7fe601c2a700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 0x7fe5e806b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:42.104 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.102+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe5f804da00 con 0x7fe5fc0fcbf0 2023-12-19T19:11:42.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:41 smithi055 ceph-mon[80978]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:42.307 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.304+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5fc10f690 con 0x7fe5e8068f60 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.309+0000 7fe5f2ffd700 1 -- 172.21.15.55:0/486187214 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7fe5fc10f690 con 0x7fe5e8068f60 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:crash 2/2 2m ago 5m * 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:mgr 2/2 2m ago 5m count:2 2023-12-19T19:11:42.311 INFO:teuthology.orchestra.run.smithi055.stdout:mon 2/2 2m ago 5m smithi055:172.21.15.55=smithi055;smithi187:172.21.15.187=smithi187;count:2 2023-12-19T19:11:42.312 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2023-12-19T19:11:42.312 INFO:teuthology.orchestra.run.smithi055.stdout:osd 6 2m ago - 2023-12-19T19:11:42.312 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 msgr2=0x7fe5e806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 0x7fe5e806b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec000c00 tx=0x7fe5ec009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 msgr2=0x7fe5fc10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe5f800eb70 tx=0x7fe5f800b040 comp rx=0 tx=0).stop 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 shutdown_connections 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe5e8068f60 0x7fe5e806b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe5fc0fcbf0 0x7fe5fc10d490 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.313 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 --2- 172.21.15.55:0/486187214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5fc0fde00 0x7fe5fc10d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:42.314 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 >> 172.21.15.55:0/486187214 conn(0x7fe5fc0f82d0 msgr2=0x7fe5fc101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:42.314 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 shutdown_connections 2023-12-19T19:11:42.314 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:42.311+0000 7fe60468f700 1 -- 172.21.15.55:0/486187214 wait complete. 2023-12-19T19:11:42.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:41 smithi187 ceph-mon[87698]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:42.903 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph orch host ls' 2023-12-19T19:11:43.170 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:43.909 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:43 smithi055 ceph-mon[80978]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:43.910 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:43 smithi055 ceph-mon[80978]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:44.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:43 smithi187 ceph-mon[87698]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:44.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:43 smithi187 ceph-mon[87698]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:44.805 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 -- 172.21.15.55:0/4105860519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 msgr2=0x7f03c80fd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/4105860519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c80fd500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03bc004940 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 -- 172.21.15.55:0/4105860519 shutdown_connections 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/4105860519 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 0x7f03c80fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/4105860519 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c80fd500 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 -- 172.21.15.55:0/4105860519 >> 172.21.15.55:0/4105860519 conn(0x7f03c80f69d0 msgr2=0x7f03c80f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 -- 172.21.15.55:0/4105860519 shutdown_connections 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.803+0000 7f03cfa6b700 1 -- 172.21.15.55:0/4105860519 wait complete. 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 Processor -- start 2023-12-19T19:11:44.806 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 -- start start 2023-12-19T19:11:44.807 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 0x7f03c810ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8110630 con 0x7f03c80fd0f0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cfa6b700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8110e70 con 0x7f03c80fda40 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33098/0 (socket says 172.21.15.55:33098) 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 -- 172.21.15.55:0/648580366 learned_addr learned my addr 172.21.15.55:0/648580366 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd006700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 0x7f03c810ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 -- 172.21.15.55:0/648580366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 msgr2=0x7f03c810ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 0x7f03c810ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 -- 172.21.15.55:0/648580366 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc005040 con 0x7f03c80fd0f0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.804+0000 7f03cd807700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009380 tx=0x7f03bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.805+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc00e040 con 0x7f03c80fd0f0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.805+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f03bc00bda0 con 0x7f03c80fd0f0 2023-12-19T19:11:44.808 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.805+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c81110f0 con 0x7f03c80fd0f0 2023-12-19T19:11:44.809 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.805+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03bc01c630 con 0x7f03c80fd0f0 2023-12-19T19:11:44.809 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.805+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c8111630 con 0x7f03c80fd0f0 2023-12-19T19:11:44.809 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.806+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804bb80 con 0x7f03c80fd0f0 2023-12-19T19:11:44.810 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.806+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f03bc0139d0 con 0x7f03c80fd0f0 2023-12-19T19:11:44.810 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.807+0000 7f03ba7fc700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 0x7f03b406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:44.810 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.807+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03bc087df0 con 0x7f03c80fd0f0 2023-12-19T19:11:44.810 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.807+0000 7f03cd006700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 0x7f03b406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:44.810 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.807+0000 7f03cd006700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 0x7f03b406b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006fd0 tx=0x7f03c4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:44.813 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:44.811+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f03bc058130 con 0x7f03c80fd0f0 2023-12-19T19:11:45.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.014+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f03c81046c0 con 0x7f03b4068f60 2023-12-19T19:11:45.018 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.015+0000 7f03ba7fc700 1 -- 172.21.15.55:0/648580366 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f03c81046c0 con 0x7f03b4068f60 2023-12-19T19:11:45.018 INFO:teuthology.orchestra.run.smithi055.stdout:HOST ADDR LABELS STATUS 2023-12-19T19:11:45.018 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055 172.21.15.55 2023-12-19T19:11:45.018 INFO:teuthology.orchestra.run.smithi055.stdout:smithi187 172.21.15.187 2023-12-19T19:11:45.018 INFO:teuthology.orchestra.run.smithi055.stdout:2 hosts in cluster 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 msgr2=0x7f03b406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 0x7f03b406b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f03c4006fd0 tx=0x7f03c4008040 comp rx=0 tx=0).stop 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 msgr2=0x7f03c810fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03bc009380 tx=0x7f03bc009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 shutdown_connections 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f03b4068f60 0x7f03b406b410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f03c80fd0f0 0x7f03c810fa60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.017+0000 7f03cfa6b700 1 --2- 172.21.15.55:0/648580366 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03c80fda40 0x7f03c810ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.018+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 >> 172.21.15.55:0/648580366 conn(0x7f03c80f69d0 msgr2=0x7f03c80fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:45.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.018+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 shutdown_connections 2023-12-19T19:11:45.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:45.018+0000 7f03cfa6b700 1 -- 172.21.15.55:0/648580366 wait complete. 2023-12-19T19:11:45.754 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph orch device ls' 2023-12-19T19:11:46.021 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:46.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:45 smithi055 ceph-mon[80978]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:46.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:45 smithi055 ceph-mon[80978]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:46.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:45 smithi187 ceph-mon[87698]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:46.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:45 smithi187 ceph-mon[87698]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/3163001101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 msgr2=0x7fe1bc0fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/3163001101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc0fd000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009ce0 tx=0x7fe1ac004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/3163001101 shutdown_connections 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/3163001101 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 0x7fe1bc0fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/3163001101 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc0fd000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/3163001101 >> 172.21.15.55:0/3163001101 conn(0x7fe1bc0f82d0 msgr2=0x7fe1bc0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/3163001101 shutdown_connections 2023-12-19T19:11:47.536 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/3163001101 wait complete. 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 Processor -- start 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.532+0000 7fe1c0dd3700 1 -- start start 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1c0dd3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1c0dd3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 0x7fe1bc06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1c0dd3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc070590 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1c0dd3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc070700 con 0x7fe1bc0fde00 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:33122/0 (socket says 172.21.15.55:33122) 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 -- 172.21.15.55:0/1530492901 learned_addr learned my addr 172.21.15.55:0/1530492901 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1b9d9b700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 0x7fe1bc06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 -- 172.21.15.55:0/1530492901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 msgr2=0x7fe1bc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 0x7fe1bc06ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 -- 172.21.15.55:0/1530492901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac009940 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.537 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1ba59c700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009880 tx=0x7fe1ac004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:47.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac00e040 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1ac0128c0 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1ac01c8b0 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.533+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc070980 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.538 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.534+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc115750 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.535+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc04bb80 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.535+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe1ac012a30 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.535+0000 7fe1aaffd700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 0x7fe1a406b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.536+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe1ac0888f0 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.536+0000 7fe1b9d9b700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 0x7fe1a406b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:47.539 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.536+0000 7fe1b9d9b700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 0x7fe1a406b470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0007fd0 tx=0x7fe1b0009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:47.542 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.540+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe1ac058a70 con 0x7fe1bc0fcbf0 2023-12-19T19:11:47.749 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.746+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1bc102750 con 0x7fe1a4068fc0 2023-12-19T19:11:47.750 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.748+0000 7fe1aaffd700 1 -- 172.21.15.55:0/1530492901 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fe1bc102750 con 0x7fe1a4068fc0 2023-12-19T19:11:47.750 INFO:teuthology.orchestra.run.smithi055.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-12-19T19:11:47.751 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102VS4P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2023-12-19T19:11:47.751 INFO:teuthology.orchestra.run.smithi055.stdout:smithi187 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500MW400BGN 372G No 73s ago Has a FileSystem, LVM detected 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 msgr2=0x7fe1a406b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 0x7fe1a406b470 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0007fd0 tx=0x7fe1b0009040 comp rx=0 tx=0).stop 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 msgr2=0x7fe1bc071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009880 tx=0x7fe1ac004d10 comp rx=0 tx=0).stop 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 shutdown_connections 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe1a4068fc0 0x7fe1a406b470 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe1bc0fcbf0 0x7fe1bc071b10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 --2- 172.21.15.55:0/1530492901 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe1bc0fde00 0x7fe1bc06ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.751+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 >> 172.21.15.55:0/1530492901 conn(0x7fe1bc0f82d0 msgr2=0x7fe1bc101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.752+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 shutdown_connections 2023-12-19T19:11:47.754 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:47.752+0000 7fe1c0dd3700 1 -- 172.21.15.55:0/1530492901 wait complete. 2023-12-19T19:11:48.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:47 smithi055 ceph-mon[80978]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:48.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:11:48.336 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:11:48.337 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2023-12-19T19:11:48.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:47 smithi187 ceph-mon[87698]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:48.604 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:49.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:48 smithi055 ceph-mon[80978]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:49.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:48 smithi187 ceph-mon[87698]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- 172.21.15.55:0/4259612153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 msgr2=0x7fbbb81003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 --2- 172.21.15.55:0/4259612153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb81003e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbba4004940 tx=0x7fbba4009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- 172.21.15.55:0/4259612153 shutdown_connections 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 --2- 172.21.15.55:0/4259612153 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb81003e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 --2- 172.21.15.55:0/4259612153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 0x7fbbb80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- 172.21.15.55:0/4259612153 >> 172.21.15.55:0/4259612153 conn(0x7fbbb80fa5d0 msgr2=0x7fbbb80fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:50.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- 172.21.15.55:0/4259612153 shutdown_connections 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- 172.21.15.55:0/4259612153 wait complete. 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 Processor -- start 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.197+0000 7fbbbf133700 1 -- start start 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 0x7fbbb810f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb81102f0 con 0x7fbbb80fff70 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb8110b90 con 0x7fbbb80fee00 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48798/0 (socket says 172.21.15.55:48798) 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 -- 172.21.15.55:0/57183642 learned_addr learned my addr 172.21.15.55:0/57183642 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbcecf700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 0x7fbbb810f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:50.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 -- 172.21.15.55:0/57183642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 msgr2=0x7fbbb810f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 0x7fbbb810f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 -- 172.21.15.55:0/57183642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4005040 con 0x7fbbb80fff70 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb7fff700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009fd0 tx=0x7fbba4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba400e040 con 0x7fbbb80fff70 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb8110e10 con 0x7fbbb80fff70 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbba4004290 con 0x7fbbb80fff70 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbba401c3f0 con 0x7fbbb80fff70 2023-12-19T19:11:50.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.198+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb8111350 con 0x7fbbb80fff70 2023-12-19T19:11:50.204 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.200+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb804bb80 con 0x7fbbb80fff70 2023-12-19T19:11:50.204 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:49 smithi055 ceph-mon[80978]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:50.204 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:50.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.200+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbba400bc50 con 0x7fbbb80fff70 2023-12-19T19:11:50.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.200+0000 7fbbb57fa700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:50.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.200+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbba4087e40 con 0x7fbbb80fff70 2023-12-19T19:11:50.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.200+0000 7fbbbcecf700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:50.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.201+0000 7fbbbcecf700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 0x7fbba806b420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0041d0 tx=0x7fbbac0043c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:50.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.205+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbba40580e0 con 0x7fbbb80fff70 2023-12-19T19:11:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:49 smithi187 ceph-mon[87698]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:11:50.431 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:50.429+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbb81116e0 con 0x7fbba8068f70 2023-12-19T19:11:51.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:50 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-19T19:11:51.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:50 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-19T19:11:52.004 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.001+0000 7fbbb57fa700 1 -- 172.21.15.55:0/57183642 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbbb81116e0 con 0x7fbba8068f70 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 msgr2=0x7fbba806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 0x7fbba806b420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0041d0 tx=0x7fbbac0043c0 comp rx=0 tx=0).stop 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 msgr2=0x7fbbb810fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009fd0 tx=0x7fbba4009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 shutdown_connections 2023-12-19T19:11:52.007 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fbba8068f70 0x7fbba806b420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:52.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbbb80fee00 0x7fbbb810f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:52.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 --2- 172.21.15.55:0/57183642 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fbbb80fff70 0x7fbbb810fcf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:52.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.004+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 >> 172.21.15.55:0/57183642 conn(0x7fbbb80fa5d0 msgr2=0x7fbbb8103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:52.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.005+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 shutdown_connections 2023-12-19T19:11:52.008 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:52.005+0000 7fbbbf133700 1 -- 172.21.15.55:0/57183642 wait complete. 2023-12-19T19:11:52.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-mon[80978]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:52.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-mon[80978]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:52.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-19T19:11:52.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-mon[80978]: osdmap e42: 6 total, 6 up, 6 in 2023-12-19T19:11:52.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-19T19:11:52.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:11:51.981+0000 7f7e9063c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T19:11:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:51 smithi187 ceph-mon[87698]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:11:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:51 smithi187 ceph-mon[87698]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:51 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-19T19:11:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:51 smithi187 ceph-mon[87698]: osdmap e42: 6 total, 6 up, 6 in 2023-12-19T19:11:52.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:51 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-19T19:11:52.693 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: osdmap e43: 6 total, 6 up, 6 in 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-19T19:11:53.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: osdmap e44: 6 total, 6 up, 6 in 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: fsmap cephfs:0 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: Saving service mds.cephfs spec with placement count:4 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.wupdfj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.wupdfj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:53.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:52 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: osdmap e43: 6 total, 6 up, 6 in 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-19T19:11:53.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: osdmap e44: 6 total, 6 up, 6 in 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: fsmap cephfs:0 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: Saving service mds.cephfs spec with placement count:4 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.wupdfj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:53.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.wupdfj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:53.343 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:52 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:53.935 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:54.294 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:53 smithi055 ceph-mon[80978]: Deploying daemon mds.cephfs.smithi055.wupdfj on smithi055 2023-12-19T19:11:54.294 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:53 smithi055 ceph-mon[80978]: pgmap v176: 65 pgs: 15 active+clean, 17 creating+peering, 33 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:54.294 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:53 smithi055 ceph-mon[80978]: osdmap e45: 6 total, 6 up, 6 in 2023-12-19T19:11:54.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:53 smithi187 ceph-mon[87698]: Deploying daemon mds.cephfs.smithi055.wupdfj on smithi055 2023-12-19T19:11:54.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:53 smithi187 ceph-mon[87698]: pgmap v176: 65 pgs: 15 active+clean, 17 creating+peering, 33 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:54.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:53 smithi187 ceph-mon[87698]: osdmap e45: 6 total, 6 up, 6 in 2023-12-19T19:11:55.220 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:55 smithi055 ceph-mon[80978]: osdmap e46: 6 total, 6 up, 6 in 2023-12-19T19:11:55.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:55 smithi187 ceph-mon[87698]: osdmap e46: 6 total, 6 up, 6 in 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 -- 172.21.15.55:0/4053825986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 msgr2=0x7f77dc0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 --2- 172.21.15.55:0/4053825986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc0fd150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f77d8004940 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 -- 172.21.15.55:0/4053825986 shutdown_connections 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 --2- 172.21.15.55:0/4053825986 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc0fd150 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 --2- 172.21.15.55:0/4053825986 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 0x7f77dc0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.207 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 -- 172.21.15.55:0/4053825986 >> 172.21.15.55:0/4053825986 conn(0x7f77dc0f7190 msgr2=0x7f77dc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 -- 172.21.15.55:0/4053825986 shutdown_connections 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.203+0000 7f77e33bf700 1 -- 172.21.15.55:0/4053825986 wait complete. 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 Processor -- start 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 -- start start 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 0x7f77dc10d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc10e130 con 0x7f77dc0fcd00 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e33bf700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc10e970 con 0x7f77dc0fbaf0 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:48830/0 (socket says 172.21.15.55:48830) 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 -- 172.21.15.55:0/2066887971 learned_addr learned my addr 172.21.15.55:0/2066887971 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e115b700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 0x7f77dc10d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 -- 172.21.15.55:0/2066887971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 msgr2=0x7f77dc10d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 0x7f77dc10d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.208 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 -- 172.21.15.55:0/2066887971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8005040 con 0x7f77dc0fcd00 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.204+0000 7f77e095a700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009380 tx=0x7f77d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.205+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d800e040 con 0x7f77dc0fcd00 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.205+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77d800bda0 con 0x7f77dc0fcd00 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.205+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc10ebf0 con 0x7f77dc0fcd00 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.205+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77d801c630 con 0x7f77dc0fcd00 2023-12-19T19:11:56.209 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.205+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77dc10f130 con 0x7f77dc0fcd00 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.206+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77dc04bb80 con 0x7f77dc0fcd00 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.207+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f77d80139d0 con 0x7f77dc0fcd00 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.207+0000 7f77cdffb700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 0x7f77c806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.207+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f77d8088130 con 0x7f77dc0fcd00 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.207+0000 7f77e115b700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 0x7f77c806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:56.210 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.208+0000 7f77e115b700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 0x7f77c806b410 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f77d0006fd0 tx=0x7f77d0008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:56.214 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.212+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f77d8058130 con 0x7f77dc0fcd00 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: pgmap v179: 65 pgs: 33 active+clean, 17 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.uciyzu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.uciyzu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:56 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: pgmap v179: 65 pgs: 33 active+clean, 17 creating+peering, 15 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.uciyzu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.uciyzu", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:56 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.451+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f77dc105620 con 0x7f77dc0fcd00 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.452+0000 7f77cdffb700 1 -- 172.21.15.55:0/2066887971 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7f77d8057cc0 con 0x7f77dc0fcd00 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stdout:e4 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stdout:legacy client fscid: 1 2023-12-19T19:11:56.454 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:Filesystem 'cephfs' (1) 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:fs_name cephfs 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 4 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:created 2023-12-19T19:11:51.982744+0000 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2023-12-19T19:11:56.021142+0000 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:tableserver 0 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:root 0 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:session_timeout 60 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:session_autoclose 300 2023-12-19T19:11:56.455 INFO:teuthology.orchestra.run.smithi055.stdout:max_file_size 1099511627776 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:required_client_features {} 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure 0 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure_osd_epoch 0 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:max_mds 1 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:in 0 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:up {0=14462} 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:failed 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:damaged 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:stopped 2023-12-19T19:11:56.456 INFO:teuthology.orchestra.run.smithi055.stdout:data_pools [3] 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout:metadata_pool 2 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout:inline_data disabled 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout:balancer 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout:standby_count_wanted 0 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi055.wupdfj{0:14462} state up:creating seq 1 addr [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:56.457 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.454+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 msgr2=0x7f77c806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.454+0000 7f77e33bf700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 0x7f77c806b410 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f77d0006fd0 tx=0x7f77d0008040 comp rx=0 tx=0).stop 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.454+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 msgr2=0x7f77dc10daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.454+0000 7f77e33bf700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009380 tx=0x7f77d8009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 shutdown_connections 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f77c8068f60 0x7f77c806b410 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.458 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77dc0fbaf0 0x7f77dc10d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.459 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 --2- 172.21.15.55:0/2066887971 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f77dc0fcd00 0x7f77dc10daa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:56.459 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 >> 172.21.15.55:0/2066887971 conn(0x7f77dc0f7190 msgr2=0x7f77dc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:56.459 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 shutdown_connections 2023-12-19T19:11:56.459 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:56.455+0000 7f77e33bf700 1 -- 172.21.15.55:0/2066887971 wait complete. 2023-12-19T19:11:56.459 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 4 2023-12-19T19:11:57.000 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:11:57.010 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:11:57.010 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2023-12-19T19:11:57.033 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: Deploying daemon mds.cephfs.smithi187.uciyzu on smithi187 2023-12-19T19:11:57.034 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:boot 2023-12-19T19:11:57.034 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: daemon mds.cephfs.smithi055.wupdfj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-19T19:11:57.034 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-19T19:11:57.034 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-19T19:11:57.283 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: Deploying daemon mds.cephfs.smithi187.uciyzu on smithi187 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:boot 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: daemon mds.cephfs.smithi055.wupdfj assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-19T19:11:57.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: Cluster is now healthy 2023-12-19T19:11:57.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: fsmap cephfs:0 1 up:standby 2023-12-19T19:11:57.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.wupdfj"}]: dispatch 2023-12-19T19:11:57.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:creating} 2023-12-19T19:11:57.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: daemon mds.cephfs.smithi055.wupdfj is now active in filesystem cephfs as rank 0 2023-12-19T19:11:57.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:57 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2066887971' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T19:11:57.309 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: Cluster is now healthy 2023-12-19T19:11:57.310 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: fsmap cephfs:0 1 up:standby 2023-12-19T19:11:57.310 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.wupdfj"}]: dispatch 2023-12-19T19:11:57.310 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:creating} 2023-12-19T19:11:57.310 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: daemon mds.cephfs.smithi055.wupdfj is now active in filesystem cephfs as rank 0 2023-12-19T19:11:57.310 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:57 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2066887971' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: pgmap v180: 65 pgs: 51 active+clean, 14 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 208 B/s wr, 0 op/s 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:active 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.187:6824/191360506,v1:172.21.15.187:6825/191360506] up:boot 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 1 up:standby 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.uciyzu"}]: dispatch 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.306 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.qvunsw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:58.307 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.qvunsw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:58.307 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:58 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:58.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: pgmap v180: 65 pgs: 51 active+clean, 14 creating+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 208 B/s wr, 0 op/s 2023-12-19T19:11:58.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:active 2023-12-19T19:11:58.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.187:6824/191360506,v1:172.21.15.187:6825/191360506] up:boot 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 1 up:standby 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.uciyzu"}]: dispatch 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.qvunsw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi055.qvunsw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:11:58.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:58 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:11:59.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:11:59 smithi055 ceph-mon[80978]: Deploying daemon mds.cephfs.smithi055.qvunsw on smithi055 2023-12-19T19:11:59.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:11:59 smithi187 ceph-mon[87698]: Deploying daemon mds.cephfs.smithi055.qvunsw on smithi055 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- 172.21.15.55:0/2900763559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89440704f0 msgr2=0x7f8944070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 --2- 172.21.15.55:0/2900763559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89440704f0 0x7f8944070960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8938004940 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- 172.21.15.55:0/2900763559 shutdown_connections 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 --2- 172.21.15.55:0/2900763559 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f89440704f0 0x7f8944070960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 --2- 172.21.15.55:0/2900763559 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8944071b40 0x7f894406ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- 172.21.15.55:0/2900763559 >> 172.21.15.55:0/2900763559 conn(0x7f89440f82d0 msgr2=0x7f89440fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- 172.21.15.55:0/2900763559 shutdown_connections 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- 172.21.15.55:0/2900763559 wait complete. 2023-12-19T19:11:59.491 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 Processor -- start 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.487+0000 7f8948f6f700 1 -- start start 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89440704f0 0x7f894410f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89441103c0 con 0x7f8944071b40 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8944110c60 con 0x7f89440704f0 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35454/0 (socket says 172.21.15.55:35454) 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 -- 172.21.15.55:0/689589416 learned_addr learned my addr 172.21.15.55:0/689589416 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f894259c700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89440704f0 0x7f894410f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 -- 172.21.15.55:0/689589416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89440704f0 msgr2=0x7f894410f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89440704f0 0x7f894410f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 -- 172.21.15.55:0/689589416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8938005040 con 0x7f8944071b40 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8941d9b700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8938009380 tx=0x7f8938009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893800e040 con 0x7f8944071b40 2023-12-19T19:11:59.492 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8944110ee0 con 0x7f8944071b40 2023-12-19T19:11:59.493 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f893800bd20 con 0x7f8944071b40 2023-12-19T19:11:59.493 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f893801c630 con 0x7f8944071b40 2023-12-19T19:11:59.493 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.488+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8944111420 con 0x7f8944071b40 2023-12-19T19:11:59.493 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.489+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89441096e0 con 0x7f8944071b40 2023-12-19T19:11:59.494 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.490+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8938013420 con 0x7f8944071b40 2023-12-19T19:11:59.494 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.490+0000 7f8932ffd700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 0x7f892c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:11:59.494 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.491+0000 7f894259c700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 0x7f892c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:11:59.494 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.491+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8938088090 con 0x7f8944071b40 2023-12-19T19:11:59.494 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.491+0000 7f894259c700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 0x7f892c06b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8934007fd0 tx=0x7f8934009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:11:59.497 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.495+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8938058090 con 0x7f8944071b40 2023-12-19T19:11:59.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:11:59.739+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f8944107870 con 0x7f8944071b40 2023-12-19T19:12:00.058 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.055+0000 7f8932ffd700 1 -- 172.21.15.55:0/689589416 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8938057c20 con 0x7f8944071b40 2023-12-19T19:12:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 msgr2=0x7f892c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 0x7f892c06b410 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8934007fd0 tx=0x7f8934009040 comp rx=0 tx=0).stop 2023-12-19T19:12:00.060 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 msgr2=0x7f894410fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8938009380 tx=0x7f8938009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 shutdown_connections 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f892c068f60 0x7f892c06b410 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89440704f0 0x7f894410f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 --2- 172.21.15.55:0/689589416 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f8944071b40 0x7f894410fdc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 >> 172.21.15.55:0/689589416 conn(0x7f89440f82d0 msgr2=0x7f8944103390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 shutdown_connections 2023-12-19T19:12:00.061 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:00.058+0000 7f8948f6f700 1 -- 172.21.15.55:0/689589416 wait complete. 2023-12-19T19:12:00.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:00 smithi055 ceph-mon[80978]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T19:12:00.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:00 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/689589416' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T19:12:00.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:00 smithi187 ceph-mon[87698]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T19:12:00.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:00 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/689589416' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-19T19:12:00.723 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:12:00.733 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:12:00.733 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2023-12-19T19:12:01.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:01 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/689589416' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-19T19:12:01.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:01 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 1 up:standby 2023-12-19T19:12:01.217 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:01 smithi055 ceph-mon[80978]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T19:12:01.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:01 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/689589416' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-19T19:12:01.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:01 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 1 up:standby 2023-12-19T19:12:01.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:01 smithi187 ceph-mon[87698]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2023-12-19T19:12:01.400 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:02.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.590 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.xdgtcm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.xdgtcm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: Deploying daemon mds.cephfs.smithi187.xdgtcm on smithi187 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.55:6828/58770560,v1:172.21.15.55:6829/58770560] up:boot 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2 up:standby 2023-12-19T19:12:02.591 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:02 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.qvunsw"}]: dispatch 2023-12-19T19:12:02.683 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.xdgtcm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi187.xdgtcm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: Deploying daemon mds.cephfs.smithi187.xdgtcm on smithi187 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.55:6828/58770560,v1:172.21.15.55:6829/58770560] up:boot 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2 up:standby 2023-12-19T19:12:02.684 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:02 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi055.qvunsw"}]: dispatch 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 -- 172.21.15.55:0/280290808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0ffd90 msgr2=0x7f7c9c102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/280290808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c102170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c90004940 tx=0x7f7c90009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 -- 172.21.15.55:0/280290808 shutdown_connections 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/280290808 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c102170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/280290808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c0ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 -- 172.21.15.55:0/280290808 >> 172.21.15.55:0/280290808 conn(0x7f7c9c0f7240 msgr2=0x7f7c9c0f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 -- 172.21.15.55:0/280290808 shutdown_connections 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.007+0000 7f7ca23d5700 1 -- 172.21.15.55:0/280290808 wait complete. 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 Processor -- start 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 -- start start 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c10da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c10e000 con 0x7f7c9c0668c0 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.008+0000 7f7ca23d5700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c10e8a0 con 0x7f7c9c0ffd90 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35488/0 (socket says 172.21.15.55:35488) 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 -- 172.21.15.55:0/1521860868 learned_addr learned my addr 172.21.15.55:0/1521860868 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9b7fe700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c10da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 -- 172.21.15.55:0/1521860868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0ffd90 msgr2=0x7f7c9c10da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c10da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.016 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 -- 172.21.15.55:0/1521860868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90005040 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c9bfff700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c002a80 tx=0x7f7c8c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c8c007cc0 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c10eb80 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c8c00f040 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c8c00d9e0 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.009+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c10f0c0 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.010+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c1075c0 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.011+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7c8c00dc50 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.011+0000 7f7c98ff9700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 0x7f7c8406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.011+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7c8c0865a0 con 0x7f7c9c0668c0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.011+0000 7f7c9b7fe700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 0x7f7c8406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:03.017 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.012+0000 7f7c9b7fe700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 0x7f7c8406b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c900043d0 tx=0x7f7c90009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:03.018 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.016+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7c8c04ca00 con 0x7f7c9c0668c0 2023-12-19T19:12:03.259 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.257+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7c9c04bb80 con 0x7f7c9c0668c0 2023-12-19T19:12:03.643 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.640+0000 7f7c98ff9700 1 -- 172.21.15.55:0/1521860868 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c8c056480 con 0x7f7c9c0668c0 2023-12-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.642+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 msgr2=0x7f7c8406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:03.645 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 0x7f7c8406b420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c900043d0 tx=0x7f7c90009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 msgr2=0x7f7c9c10d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c002a80 tx=0x7f7c8c0046b0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 shutdown_connections 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7c84068f70 0x7f7c8406b420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f7c9c0668c0 0x7f7c9c10d4c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 --2- 172.21.15.55:0/1521860868 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c9c0ffd90 0x7f7c9c10da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:03.646 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 >> 172.21.15.55:0/1521860868 conn(0x7f7c9c0f7240 msgr2=0x7f7c9c0f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 shutdown_connections 2023-12-19T19:12:03.647 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:03.643+0000 7f7ca23d5700 1 -- 172.21.15.55:0/1521860868 wait complete. 2023-12-19T19:12:03.957 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:03 smithi055 ceph-mon[80978]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2023-12-19T19:12:03.957 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:03 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1521860868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2023-12-19T19:12:04.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:03 smithi187 ceph-mon[87698]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2023-12-19T19:12:04.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:03 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1521860868' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2023-12-19T19:12:04.276 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:12:04.287 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:12:04.288 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2023-12-19T19:12:04.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1521860868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2023-12-19T19:12:04.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2 up:standby 2023-12-19T19:12:04.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:04.840 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:04.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:04.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:04.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:04.841 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:12:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1521860868' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2023-12-19T19:12:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 2 up:standby 2023-12-19T19:12:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:05.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:05.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:12:05.116 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.187:6826/1691713828,v1:172.21.15.187:6827/1691713828] up:boot 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:active 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.xdgtcm"}]: dispatch 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:05.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:05 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:06.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.187:6826/1691713828,v1:172.21.15.187:6827/1691713828] up:boot 2023-12-19T19:12:06.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] up:active 2023-12-19T19:12:06.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:06.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi187.xdgtcm"}]: dispatch 2023-12-19T19:12:06.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 4 op/s 2023-12-19T19:12:06.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:06.091 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:05 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:07.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:06 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.55:6828/58770560,v1:172.21.15.55:6829/58770560] up:standby 2023-12-19T19:12:07.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:06 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.187:6824/191360506,v1:172.21.15.187:6825/191360506] up:standby 2023-12-19T19:12:07.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:06 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:07.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:07.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:06 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 -- 172.21.15.55:0/1395518082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fcbf0 msgr2=0x7f22480fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 --2- 172.21.15.55:0/1395518082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fcbf0 0x7f22480fd000 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f223c004940 tx=0x7f223c009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 -- 172.21.15.55:0/1395518082 shutdown_connections 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 --2- 172.21.15.55:0/1395518082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fde00 0x7f22480fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 --2- 172.21.15.55:0/1395518082 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fcbf0 0x7f22480fd000 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.286+0000 7f224f14c700 1 -- 172.21.15.55:0/1395518082 >> 172.21.15.55:0/1395518082 conn(0x7f22480f82d0 msgr2=0x7f22480fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 -- 172.21.15.55:0/1395518082 shutdown_connections 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 -- 172.21.15.55:0/1395518082 wait complete. 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 Processor -- start 2023-12-19T19:12:07.291 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 -- start start 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fcbf0 0x7f2248071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2248070590 con 0x7f22480fde00 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.287+0000 7f224f14c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2248070700 con 0x7f22480fcbf0 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35502/0 (socket says 172.21.15.55:35502) 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 -- 172.21.15.55:0/1480405529 learned_addr learned my addr 172.21.15.55:0/1480405529 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f224cee8700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fcbf0 0x7f2248071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 -- 172.21.15.55:0/1480405529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fcbf0 msgr2=0x7f2248071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fcbf0 0x7f2248071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 -- 172.21.15.55:0/1480405529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c005040 con 0x7f22480fde00 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f2247fff700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f223400eb40 tx=0x7f223400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:07.292 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2234007cc0 con 0x7f22480fde00 2023-12-19T19:12:07.293 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f223400f040 con 0x7f22480fde00 2023-12-19T19:12:07.293 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2248070980 con 0x7f22480fde00 2023-12-19T19:12:07.293 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f223400c990 con 0x7f22480fde00 2023-12-19T19:12:07.293 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.288+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2248115500 con 0x7f22480fde00 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.289+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224804bb80 con 0x7f22480fde00 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.290+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f223400caf0 con 0x7f22480fde00 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.290+0000 7f22457fa700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 0x7f223806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.291+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f22340868d0 con 0x7f22480fde00 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.291+0000 7f224cee8700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 0x7f223806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:07.294 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.291+0000 7f224cee8700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 0x7f223806b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f223c009fd0 tx=0x7f223c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:07.297 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.295+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f223404da00 con 0x7f22480fde00 2023-12-19T19:12:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:06 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.55:6828/58770560,v1:172.21.15.55:6829/58770560] up:standby 2023-12-19T19:12:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:06 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.187:6824/191360506,v1:172.21.15.187:6825/191360506] up:standby 2023-12-19T19:12:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:06 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:07.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:07.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:06 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:07.545 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:07.543+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2248105590 con 0x7f22480fde00 2023-12-19T19:12:08.020 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.017+0000 7f22457fa700 1 -- 172.21.15.55:0/1480405529 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7f2248105590 con 0x7f22480fde00 2023-12-19T19:12:08.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 msgr2=0x7f223806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:08.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 0x7f223806b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f223c009fd0 tx=0x7f223c009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:08.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 msgr2=0x7f224806ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:08.021 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f223400eb40 tx=0x7f223400b040 comp rx=0 tx=0).stop 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 shutdown_connections 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2238068f60 0x7f223806b410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f22480fcbf0 0x7f2248071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 --2- 172.21.15.55:0/1480405529 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f22480fde00 0x7f224806ff90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 >> 172.21.15.55:0/1480405529 conn(0x7f22480f82d0 msgr2=0x7f2248101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 shutdown_connections 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:08.019+0000 7f224f14c700 1 -- 172.21.15.55:0/1480405529 wait complete. 2023-12-19T19:12:08.022 INFO:teuthology.orchestra.run.smithi055.stderr:inline data enabled 2023-12-19T19:12:08.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:08 smithi055 ceph-mon[80978]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2023-12-19T19:12:08.300 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:08 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1480405529' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-19T19:12:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:08 smithi187 ceph-mon[87698]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2023-12-19T19:12:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:08 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1480405529' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-19T19:12:09.021 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-19T19:12:09.031 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:12:09.032 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-19T19:12:09.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:09 smithi055 ceph-mon[80978]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-19T19:12:09.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:09 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1480405529' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-19T19:12:09.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:09 smithi055 ceph-mon[80978]: mds.? [v2:172.21.15.187:6826/1691713828,v1:172.21.15.187:6827/1691713828] up:standby 2023-12-19T19:12:09.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:09 smithi055 ceph-mon[80978]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:09.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:09 smithi187 ceph-mon[87698]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-19T19:12:09.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:09 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1480405529' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-19T19:12:09.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:09 smithi187 ceph-mon[87698]: mds.? [v2:172.21.15.187:6826/1691713828,v1:172.21.15.187:6827/1691713828] up:standby 2023-12-19T19:12:09.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:09 smithi187 ceph-mon[87698]: fsmap cephfs:1 {0=cephfs.smithi055.wupdfj=up:active} 3 up:standby 2023-12-19T19:12:09.418 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:10.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:10 smithi055 ceph-mon[80978]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2023-12-19T19:12:10.309 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:10 smithi187 ceph-mon[87698]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.482+0000 7fcceff7c700 1 -- 172.21.15.55:0/2975528543 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce80ffd80 msgr2=0x7fcce81021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.482+0000 7fcceff7c700 1 --2- 172.21.15.55:0/2975528543 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce80ffd80 0x7fcce81021e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fccd8004940 tx=0x7fccd8009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.482+0000 7fcceff7c700 1 -- 172.21.15.55:0/2975528543 shutdown_connections 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 --2- 172.21.15.55:0/2975528543 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce80ffd80 0x7fcce81021e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 --2- 172.21.15.55:0/2975528543 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce8066920 0x7fcce80ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- 172.21.15.55:0/2975528543 >> 172.21.15.55:0/2975528543 conn(0x7fcce80f71c0 msgr2=0x7fcce80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- 172.21.15.55:0/2975528543 shutdown_connections 2023-12-19T19:12:11.486 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- 172.21.15.55:0/2975528543 wait complete. 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 Processor -- start 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- start start 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 0x7fcce80fb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80fbcb0 con 0x7fcce8066920 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.483+0000 7fcceff7c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80fbe20 con 0x7fcce80ffd80 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44204/0 (socket says 172.21.15.55:44204) 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 -- 172.21.15.55:0/3984781044 learned_addr learned my addr 172.21.15.55:0/3984781044 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fcced517700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 0x7fcce80fb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 -- 172.21.15.55:0/3984781044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 msgr2=0x7fcce80fb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 0x7fcce80fb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 -- 172.21.15.55:0/3984781044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8005040 con 0x7fcce8066920 2023-12-19T19:12:11.487 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fcced517700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 0x7fcce80fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-19T19:12:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccedd18700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcce4002a50 tx=0x7fcce40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce4007cc0 con 0x7fcce8066920 2023-12-19T19:12:11.488 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcce400f040 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce80fc100 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcce4009540 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.484+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce8115550 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.485+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce81075d0 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.486+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcce40096a0 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.486+0000 7fccde7fc700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 0x7fccd406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.487+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcce4086ac0 con 0x7fcce8066920 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.487+0000 7fcced517700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 0x7fccd406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:11.489 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.487+0000 7fcced517700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 0x7fccd406b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b3b0 tx=0x7fccd8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:11.493 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.491+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcce4056a20 con 0x7fcce8066920 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.790+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fcce8105590 con 0x7fcce8066920 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.790+0000 7fccde7fc700 1 -- 172.21.15.55:0/3984781044 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 12 v12) v1 ==== 76+0+1804 (secure 0 0 0) 0x7fcce40565b0 con 0x7fcce8066920 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:e12 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:legacy client fscid: 1 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:Filesystem 'cephfs' (1) 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:fs_name cephfs 2023-12-19T19:12:11.793 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 12 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:created 2023-12-19T19:11:51.982744+0000 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:modified 2023-12-19T19:12:08.014106+0000 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:tableserver 0 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:root 0 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:session_timeout 60 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:session_autoclose 300 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:max_file_size 1099511627776 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:required_client_features {} 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure 0 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:last_failure_osd_epoch 0 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:max_mds 1 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:in 0 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:up {0=14462} 2023-12-19T19:12:11.794 INFO:teuthology.orchestra.run.smithi055.stdout:failed 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:damaged 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:stopped 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:data_pools [3] 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:metadata_pool 2 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:inline_data enabled 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:balancer 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:standby_count_wanted 1 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi055.wupdfj{0:14462} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.55:6826/3287430109,v1:172.21.15.55:6827/3287430109] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:Standby daemons: 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi055.qvunsw{-1:14474} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.55:6828/58770560,v1:172.21.15.55:6829/58770560] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi187.uciyzu{-1:24267} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.187:6824/191360506,v1:172.21.15.187:6825/191360506] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T19:12:11.795 INFO:teuthology.orchestra.run.smithi055.stdout:[mds.cephfs.smithi187.xdgtcm{-1:24281} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.187:6826/1691713828,v1:172.21.15.187:6827/1691713828] compat {c=[1],r=[1],i=[7ff]}] 2023-12-19T19:12:11.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.792+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 msgr2=0x7fccd406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:11.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.792+0000 7fcceff7c700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 0x7fccd406b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fccd800b3b0 tx=0x7fccd8009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:11.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.792+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 msgr2=0x7fcce80fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:11.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.792+0000 7fcceff7c700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcce4002a50 tx=0x7fcce40046b0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.796 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 shutdown_connections 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fccd4068f60 0x7fccd406b410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcce8066920 0x7fcce80fd230 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 --2- 172.21.15.55:0/3984781044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcce80ffd80 0x7fcce80fb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 >> 172.21.15.55:0/3984781044 conn(0x7fcce80f71c0 msgr2=0x7fcce80f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 shutdown_connections 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:11.793+0000 7fcceff7c700 1 -- 172.21.15.55:0/3984781044 wait complete. 2023-12-19T19:12:11.797 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 12 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:12:12.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:12 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3984781044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:12:12.316 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:12 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3984781044' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-19T19:12:12.424 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2023-12-19T19:12:13.352 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:13.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:13 smithi055 ceph-mon[80978]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:13 smithi187 ceph-mon[87698]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:14.953 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:14 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:14.953 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:14 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:14 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:15.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:14 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 -- 172.21.15.55:0/1054853019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fde90 msgr2=0x7fcdbc100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 --2- 172.21.15.55:0/1054853019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc100310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0004940 tx=0x7fcdb0009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 -- 172.21.15.55:0/1054853019 shutdown_connections 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 --2- 172.21.15.55:0/1054853019 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc100310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 --2- 172.21.15.55:0/1054853019 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc0fd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.366 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 -- 172.21.15.55:0/1054853019 >> 172.21.15.55:0/1054853019 conn(0x7fcdbc0f71c0 msgr2=0x7fcdbc0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 -- 172.21.15.55:0/1054853019 shutdown_connections 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.362+0000 7fcdc4108700 1 -- 172.21.15.55:0/1054853019 wait complete. 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 Processor -- start 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 -- start start 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc1140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc114740 con 0x7fcdbc0fb530 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc4108700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc114f80 con 0x7fcdbc0fde90 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:44230/0 (socket says 172.21.15.55:44230) 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 -- 172.21.15.55:0/2489533444 learned_addr learned my addr 172.21.15.55:0/2489533444 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc16a3700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc1140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 -- 172.21.15.55:0/2489533444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fde90 msgr2=0x7fcdbc1140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:15.367 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc1140b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 -- 172.21.15.55:0/2489533444 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb0005040 con 0x7fcdbc0fb530 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.363+0000 7fcdc1ea4700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400b790 tx=0x7fcdb4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.364+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb400f770 con 0x7fcdbc0fb530 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.364+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc115260 con 0x7fcdbc0fb530 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.364+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdb400fd80 con 0x7fcdbc0fb530 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.364+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdb400d310 con 0x7fcdbc0fb530 2023-12-19T19:12:15.368 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.364+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc1157a0 con 0x7fcdbc0fb530 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.365+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc04bb80 con 0x7fcdbc0fb530 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.365+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcdb4010420 con 0x7fcdbc0fb530 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.366+0000 7fcdba7fc700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 0x7fcda406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.366+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fcdb4086d80 con 0x7fcdbc0fb530 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.366+0000 7fcdc16a3700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 0x7fcda406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:15.369 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.366+0000 7fcdc16a3700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 0x7fcda406b420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0004f90 tx=0x7fcdb0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:15.372 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.370+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcdb4056ce0 con 0x7fcdbc0fb530 2023-12-19T19:12:15.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.617+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdbc10bc10 con 0x7fcdbc0fb530 2023-12-19T19:12:15.620 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.618+0000 7fcdba7fc700 1 -- 172.21.15.55:0/2489533444 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+4683 (secure 0 0 0) 0x7fcdbc10bc10 con 0x7fcdbc0fb530 2023-12-19T19:12:15.622 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 msgr2=0x7fcda406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 0x7fcda406b420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0004f90 tx=0x7fcdb0009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 msgr2=0x7fcdbc113b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcdb400b790 tx=0x7fcdb4009250 comp rx=0 tx=0).stop 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 shutdown_connections 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fcda4068f70 0x7fcda406b420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fcdbc0fb530 0x7fcdbc113b70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 --2- 172.21.15.55:0/2489533444 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcdbc0fde90 0x7fcdbc1140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 >> 172.21.15.55:0/2489533444 conn(0x7fcdbc0f71c0 msgr2=0x7fcdbc0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 shutdown_connections 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:15.620+0000 7fcdc4108700 1 -- 172.21.15.55:0/2489533444 wait complete. 2023-12-19T19:12:15.623 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 12 2023-12-19T19:12:15.633 INFO:teuthology.orchestra.run.smithi055.stdout:true 2023-12-19T19:12:16.009 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:15 smithi055 ceph-mon[80978]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:16.009 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:15 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/2489533444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T19:12:16.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:15 smithi187 ceph-mon[87698]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:16.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:15 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/2489533444' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T19:12:16.295 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2023-12-19T19:12:16.650 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:18.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:17 smithi055 ceph-mon[80978]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:18.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:17 smithi187 ceph-mon[87698]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:19.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.495+0000 7f23d3459700 1 -- 172.21.15.55:0/2235268320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 msgr2=0x7f23cc0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:19.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.495+0000 7f23d3459700 1 --2- 172.21.15.55:0/2235268320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc0fd150 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f23bc004940 tx=0x7f23bc009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:19.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 -- 172.21.15.55:0/2235268320 shutdown_connections 2023-12-19T19:12:19.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 --2- 172.21.15.55:0/2235268320 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc0fd150 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 --2- 172.21.15.55:0/2235268320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 0x7f23cc0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 -- 172.21.15.55:0/2235268320 >> 172.21.15.55:0/2235268320 conn(0x7f23cc0f71d0 msgr2=0x7f23cc0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 -- 172.21.15.55:0/2235268320 shutdown_connections 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 -- 172.21.15.55:0/2235268320 wait complete. 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 Processor -- start 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.496+0000 7f23d3459700 1 -- start start 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d3459700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 0x7f23cc112680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d3459700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d3459700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc066890 con 0x7f23cc0fcd00 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d3459700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc0669d0 con 0x7f23cc0fbaf0 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56140/0 (socket says 172.21.15.55:56140) 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 -- 172.21.15.55:0/1844977652 learned_addr learned my addr 172.21.15.55:0/1844977652 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d11f5700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 0x7f23cc112680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 -- 172.21.15.55:0/1844977652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 msgr2=0x7f23cc112680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:19.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 0x7f23cc112680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 -- 172.21.15.55:0/1844977652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc005040 con 0x7f23cc0fcd00 2023-12-19T19:12:19.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23d09f4700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009360 tx=0x7f23bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.497+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc00e040 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.498+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc115110 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.498+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23bc00bc50 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.498+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc01c3f0 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.498+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc115460 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.499+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc04bb80 con 0x7f23cc0fcd00 2023-12-19T19:12:19.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.499+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f23bc013420 con 0x7f23cc0fcd00 2023-12-19T19:12:19.503 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.500+0000 7f23c1ffb700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 0x7f23b806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:19.503 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.500+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f23bc088100 con 0x7f23cc0fcd00 2023-12-19T19:12:19.503 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.500+0000 7f23d11f5700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 0x7f23b806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:19.503 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.500+0000 7f23d11f5700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 0x7f23b806b410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23cc066620 tx=0x7f23c800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:19.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.505+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f23bc054550 con 0x7f23cc0fcd00 2023-12-19T19:12:19.825 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.823+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f23cc115680 con 0x7f23cc0fcd00 2023-12-19T19:12:19.825 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.823+0000 7f23c1ffb700 1 -- 172.21.15.55:0/1844977652 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v12) v1 ==== 78+0+99 (secure 0 0 0) 0x7f23bc057b70 con 0x7f23cc0fcd00 2023-12-19T19:12:19.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 msgr2=0x7f23b806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:19.827 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 0x7f23b806b410 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23cc066620 tx=0x7f23c800b040 comp rx=0 tx=0).stop 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 msgr2=0x7f23cc114bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009360 tx=0x7f23bc009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 shutdown_connections 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f23b8068f60 0x7f23b806b410 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f23cc0fbaf0 0x7f23cc112680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 --2- 172.21.15.55:0/1844977652 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f23cc0fcd00 0x7f23cc114bd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 >> 172.21.15.55:0/1844977652 conn(0x7f23cc0f71d0 msgr2=0x7f23cc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 shutdown_connections 2023-12-19T19:12:19.828 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:19.825+0000 7f23d3459700 1 -- 172.21.15.55:0/1844977652 wait complete. 2023-12-19T19:12:19.838 INFO:teuthology.orchestra.run.smithi055.stdout:true 2023-12-19T19:12:20.124 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1844977652' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:12:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1844977652' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-19T19:12:20.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:12:20.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:20.459 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2023-12-19T19:12:20.472 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T19:12:20.599 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- 172.21.15.55:0/689026991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 msgr2=0x7f2080101500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.599 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 --2- 172.21.15.55:0/689026991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f2080101500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f20700048d0 tx=0x7f2070009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- 172.21.15.55:0/689026991 shutdown_connections 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 --2- 172.21.15.55:0/689026991 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f2080101500 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 --2- 172.21.15.55:0/689026991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 0x7f20800febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- 172.21.15.55:0/689026991 >> 172.21.15.55:0/689026991 conn(0x7f20800f6610 msgr2=0x7f20800f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- 172.21.15.55:0/689026991 shutdown_connections 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- 172.21.15.55:0/689026991 wait complete. 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 Processor -- start 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- start start 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 0x7f208010c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.600 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208010d2e0 con 0x7f20800ff120 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f20882ae640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2080111d30 con 0x7f2080065fe0 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f2085849640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f2085849640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56144/0 (socket says 172.21.15.55:56144) 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.594+0000 7f2085849640 1 -- 172.21.15.55:0/4289231 learned_addr learned my addr 172.21.15.55:0/4289231 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f208604a640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 0x7f208010c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2085849640 1 -- 172.21.15.55:0/4289231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 msgr2=0x7f208010c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2085849640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 0x7f208010c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2085849640 1 -- 172.21.15.55:0/4289231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2070005040 con 0x7f20800ff120 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2085849640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2070034040 tx=0x7f2070009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:20.601 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20700094c0 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2070009620 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2080111e70 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f207003b640 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.595+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20801123e0 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.596+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2080104970 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.597+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f207002f070 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.597+0000 7f2076ffd640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 0x7f206c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.597+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f20700a93d0 con 0x7f20800ff120 2023-12-19T19:12:20.602 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.597+0000 7f208604a640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 0x7f206c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.603 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.597+0000 7f208604a640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 0x7f206c06b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2080065d40 tx=0x7f2078009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:20.604 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.601+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f20700758a0 con 0x7f20800ff120 2023-12-19T19:12:20.776 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.773+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2080112ae0 con 0x7f20800ff120 2023-12-19T19:12:20.776 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.774+0000 7f2076ffd640 1 -- 172.21.15.55:0/4289231 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14276 (secure 0 0 0) 0x7f2070075020 con 0x7f20800ff120 2023-12-19T19:12:20.777 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:20.777 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":46,"fsid":"63de9ba6-9ea1-11ee-95a5-87774f69a715","created":"2023-12-19T19:04:57.836701+0000","modified":"2023-12-19T19:11:53.988238+0000","last_up_change":"2023-12-19T19:10:32.592061+0000","last_in_change":"2023-12-19T19:10:13.790913+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T19:09:19.364422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2023-12-19T19:11:50.867225+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2023-12-19T19:11:51.411430+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e4ca473b-4320-4248-b6b1-fb3182ca21ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6803","nonce":3885599653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6805","nonce":3885599653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6809","nonce":3885599653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":3885599653},{"type":"v1","addr":"172.21.15.55:6807","nonce":3885599653}]},"public_addr":"172.21.15.55:6803/3885599653","cluster_addr":"172.21.15.55:6805/3885599653","heartbeat_back_addr":"172.21.15.55:6809/3885599653","heartbeat_front_addr":"172.21.15.55:6807/3885599653","state":["exists","up"]},{"osd":1,"uuid":"c34d9690-2f84-4b5d-9aab-47539ed51db7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6811","nonce":2714383701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6813","nonce":2714383701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6817","nonce":2714383701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":2714383701},{"type":"v1","addr":"172.21.15.55:6815","nonce":2714383701}]},"public_addr":"172.21.15.55:6811/2714383701","cluster_addr":"172.21.15.55:6813/2714383701","heartbeat_back_addr":"172.21.15.55:6817/2714383701","heartbeat_front_addr":"172.21.15.55:6815/2714383701","state":["exists","up"]},{"osd":2,"uuid":"a97873c3-3af6-4d1f-bd7a-d1fb5d959f7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6819","nonce":3562432537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6821","nonce":3562432537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6825","nonce":3562432537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":3562432537},{"type":"v1","addr":"172.21.15.55:6823","nonce":3562432537}]},"public_addr":"172.21.15.55:6819/3562432537","cluster_addr":"172.21.15.55:6821/3562432537","heartbeat_back_addr":"172.21.15.55:6825/3562432537","heartbeat_front_addr":"172.21.15.55:6823/3562432537","state":["exists","up"]},{"osd":3,"uuid":"b0ea8c34-b465-466e-a827-d096715d37f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6801","nonce":1951987134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6803","nonce":1951987134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6807","nonce":1951987134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":1951987134},{"type":"v1","addr":"172.21.15.187:6805","nonce":1951987134}]},"public_addr":"172.21.15.187:6801/1951987134","cluster_addr":"172.21.15.187:6803/1951987134","heartbeat_back_addr":"172.21.15.187:6807/1951987134","heartbeat_front_addr":"172.21.15.187:6805/1951987134","state":["exists","up"]},{"osd":4,"uuid":"d6d80173-9176-4bde-a702-f4a43309d5f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6809","nonce":2860720480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6811","nonce":2860720480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6815","nonce":2860720480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":2860720480},{"type":"v1","addr":"172.21.15.187:6813","nonce":2860720480}]},"public_addr":"172.21.15.187:6809/2860720480","cluster_addr":"172.21.15.187:6811/2860720480","heartbeat_back_addr":"172.21.15.187:6815/2860720480","heartbeat_front_addr":"172.21.15.187:6813/2860720480","state":["exists","up"]},{"osd":5,"uuid":"dc051ac7-0460-497e-a684-63aea173b125","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6817","nonce":1722753525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6819","nonce":1722753525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6823","nonce":1722753525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":1722753525},{"type":"v1","addr":"172.21.15.187:6821","nonce":1722753525}]},"public_addr":"172.21.15.187:6817/1722753525","cluster_addr":"172.21.15.187:6819/1722753525","heartbeat_back_addr":"172.21.15.187:6823/1722753525","heartbeat_front_addr":"172.21.15.187:6821/1722753525","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:20.662263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:08:47.762312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:16.369465+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:09:39.397498+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:05.487467+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-19T19:10:30.696534+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.55:0/1287666552":"2023-12-20T19:07:04.834074+0000","172.21.15.55:6800/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2442422185":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/3109624574":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/907073539":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2629565350":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6800/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/2922737294":"2023-12-20T19:06:02.768162+0000","172.21.15.55:0/937693294":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/2933015314":"2023-12-20T19:05:32.358078+0000","172.21.15.55:0/4084383373":"2023-12-20T19:06:02.768162+0000","172.21.15.55:6801/797433438":"2023-12-20T19:07:04.834074+0000","172.21.15.55:0/2133728457":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6800/917343204":"2023-12-20T19:05:32.358078+0000","172.21.15.55:6801/917343204":"2023-12-20T19:05:32.358078+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 msgr2=0x7f206c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 0x7f206c06b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2080065d40 tx=0x7f2078009040 comp rx=0 tx=0).stop 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 msgr2=0x7f208010cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2070034040 tx=0x7f2070009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 shutdown_connections 2023-12-19T19:12:20.779 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f206c068ed0 0x7f206c06b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.780 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f20800ff120 0x7f208010cc90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.780 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.776+0000 7f20882ae640 1 --2- 172.21.15.55:0/4289231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2080065fe0 0x7f208010c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.780 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.777+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 >> 172.21.15.55:0/4289231 conn(0x7f20800f6610 msgr2=0x7f20800f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:20.780 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.777+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 shutdown_connections 2023-12-19T19:12:20.780 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.777+0000 7f20882ae640 1 -- 172.21.15.55:0/4289231 wait complete. 2023-12-19T19:12:20.790 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-19T19:09:19.364422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2023-12-19T19:11:50.867225+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2023-12-19T19:11:51.411430+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-19T19:12:20.791 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-19T19:12:20.899 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.895+0000 7f058f6ae640 1 -- 172.21.15.55:0/1966439070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fdfc0 msgr2=0x7f05880fa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.895+0000 7f058f6ae640 1 --2- 172.21.15.55:0/1966439070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fdfc0 0x7f05880fa090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f05780048d0 tx=0x7f0578009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.895+0000 7f058f6ae640 1 -- 172.21.15.55:0/1966439070 shutdown_connections 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.895+0000 7f058f6ae640 1 --2- 172.21.15.55:0/1966439070 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fdfc0 0x7f05880fa090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.895+0000 7f058f6ae640 1 --2- 172.21.15.55:0/1966439070 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fd5e0 0x7f05880fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- 172.21.15.55:0/1966439070 >> 172.21.15.55:0/1966439070 conn(0x7f05880f5d40 msgr2=0x7f05880f8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- 172.21.15.55:0/1966439070 shutdown_connections 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- 172.21.15.55:0/1966439070 wait complete. 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 Processor -- start 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- start start 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fdfc0 0x7f058810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.900 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058810d3f0 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058f6ae640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0588111e40 con 0x7f05880fdfc0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058d44a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058d44a640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56170/0 (socket says 172.21.15.55:56170) 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.896+0000 7f058d44a640 1 -- 172.21.15.55:0/868687044 learned_addr learned my addr 172.21.15.55:0/868687044 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058d44a640 1 -- 172.21.15.55:0/868687044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fdfc0 msgr2=0x7f058810cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058cc49640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fdfc0 0x7f058810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058d44a640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fdfc0 0x7f058810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058d44a640 1 -- 172.21.15.55:0/868687044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578005040 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058d44a640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f058000eb70 tx=0x7f058000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0580007c30 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f058000f040 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0588111fe0 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f058000cb80 con 0x7f05880fd5e0 2023-12-19T19:12:20.901 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.897+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05881124f0 con 0x7f05880fd5e0 2023-12-19T19:12:20.902 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.898+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05881049e0 con 0x7f05880fd5e0 2023-12-19T19:12:20.902 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.898+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f058000cd00 con 0x7f05880fd5e0 2023-12-19T19:12:20.903 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.899+0000 7f057dffb640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 0x7f057406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:20.903 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.899+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0580086ff0 con 0x7f05880fd5e0 2023-12-19T19:12:20.903 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.899+0000 7f058cc49640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 0x7f057406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:20.903 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.899+0000 7f058cc49640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 0x7f057406b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f057802bda0 tx=0x7f0578009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:20.905 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:20.903+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f05800534c0 con 0x7f05880fd5e0 2023-12-19T19:12:21.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:20 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4289231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:12:21.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.076+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05881127c0 con 0x7f05880fd5e0 2023-12-19T19:12:21.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.076+0000 7f057dffb640 1 -- 172.21.15.55:0/868687044 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0580056ae0 con 0x7f05880fd5e0 2023-12-19T19:12:21.079 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 1 2023-12-19T19:12:21.081 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 msgr2=0x7f057406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.081 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 0x7f057406b380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f057802bda0 tx=0x7f0578009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 msgr2=0x7f058810c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f058000eb70 tx=0x7f058000b040 comp rx=0 tx=0).stop 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 shutdown_connections 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f05880fdfc0 0x7f058810cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f0574068ed0 0x7f057406b380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 --2- 172.21.15.55:0/868687044 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f05880fd5e0 0x7f058810c860 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 >> 172.21.15.55:0/868687044 conn(0x7f05880f5d40 msgr2=0x7f05881005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 shutdown_connections 2023-12-19T19:12:21.082 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.078+0000 7f058f6ae640 1 -- 172.21.15.55:0/868687044 wait complete. 2023-12-19T19:12:21.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 -- 172.21.15.55:0/419723241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 msgr2=0x7f61500fd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 --2- 172.21.15.55:0/419723241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f61500fd740 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61380048d0 tx=0x7f6138009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 -- 172.21.15.55:0/419723241 shutdown_connections 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 --2- 172.21.15.55:0/419723241 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f61500fd740 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 --2- 172.21.15.55:0/419723241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 0x7f61500fa630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 -- 172.21.15.55:0/419723241 >> 172.21.15.55:0/419723241 conn(0x7f61500f5d00 msgr2=0x7f61500f8130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 -- 172.21.15.55:0/419723241 shutdown_connections 2023-12-19T19:12:21.200 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.195+0000 7f6156d5d640 1 -- 172.21.15.55:0/419723241 wait complete. 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 Processor -- start 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 -- start start 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 0x7f615010c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f615010d2d0 con 0x7f61500fd2d0 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f6156d5d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150111d20 con 0x7f61500fa220 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f614ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f614ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56190/0 (socket says 172.21.15.55:56190) 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.196+0000 7f614ffff640 1 -- 172.21.15.55:0/3181996565 learned_addr learned my addr 172.21.15.55:0/3181996565 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f6154af9640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 0x7f615010c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614ffff640 1 -- 172.21.15.55:0/3181996565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 msgr2=0x7f615010c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.201 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614ffff640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 0x7f615010c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614ffff640 1 -- 172.21.15.55:0/3181996565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138005040 con 0x7f61500fd2d0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614ffff640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f613802bda0 tx=0x7f6138009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f61380094c0 con 0x7f61500fd2d0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6138009620 con 0x7f61500fd2d0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6150111e60 con 0x7f61500fd2d0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f613803b400 con 0x7f61500fd2d0 2023-12-19T19:12:21.202 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.197+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61501123d0 con 0x7f61500fd2d0 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.198+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6150104980 con 0x7f61500fd2d0 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.198+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f613802f070 con 0x7f61500fd2d0 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.199+0000 7f614d7fa640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 0x7f613c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.199+0000 7f6154af9640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 0x7f613c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.199+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f61380a9360 con 0x7f61500fd2d0 2023-12-19T19:12:21.203 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.199+0000 7f6154af9640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 0x7f613c06b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6150063e90 tx=0x7f614000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.205 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.203+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6138075830 con 0x7f61500fd2d0 2023-12-19T19:12:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:20 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/4289231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-19T19:12:21.376 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.373+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6150112ab0 con 0x7f61500fd2d0 2023-12-19T19:12:21.376 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.374+0000 7f614d7fa640 1 -- 172.21.15.55:0/3181996565 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6138078e50 con 0x7f61500fd2d0 2023-12-19T19:12:21.377 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 msgr2=0x7f613c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 0x7f613c06b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6150063e90 tx=0x7f614000b040 comp rx=0 tx=0).stop 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 msgr2=0x7f615010cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f613802bda0 tx=0x7f6138009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 shutdown_connections 2023-12-19T19:12:21.379 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f613c068ed0 0x7f613c06b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.380 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f61500fd2d0 0x7f615010cc80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.380 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 --2- 172.21.15.55:0/3181996565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f61500fa220 0x7f615010c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.380 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 >> 172.21.15.55:0/3181996565 conn(0x7f61500f5d00 msgr2=0x7f6150100500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.381 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 shutdown_connections 2023-12-19T19:12:21.381 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.376+0000 7f6156d5d640 1 -- 172.21.15.55:0/3181996565 wait complete. 2023-12-19T19:12:21.390 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2023-12-19T19:12:21.498 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 -- 172.21.15.55:0/2182315140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fae30 msgr2=0x7effe00fb240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 --2- 172.21.15.55:0/2182315140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fae30 0x7effe00fb240 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7effd0009c50 tx=0x7effd0004d10 comp rx=0 tx=0).stop 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 -- 172.21.15.55:0/2182315140 shutdown_connections 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 --2- 172.21.15.55:0/2182315140 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fc050 0x7effe00fc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 --2- 172.21.15.55:0/2182315140 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fae30 0x7effe00fb240 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 -- 172.21.15.55:0/2182315140 >> 172.21.15.55:0/2182315140 conn(0x7effe00f6580 msgr2=0x7effe00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 -- 172.21.15.55:0/2182315140 shutdown_connections 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.494+0000 7effe5cae640 1 -- 172.21.15.55:0/2182315140 wait complete. 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 Processor -- start 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 -- start start 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fae30 0x7effe010c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe010d320 con 0x7effe00fc050 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effe5cae640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0111d70 con 0x7effe00fae30 2023-12-19T19:12:21.499 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effdeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effdeffd640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56212/0 (socket says 172.21.15.55:56212) 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effdeffd640 1 -- 172.21.15.55:0/4202984378 learned_addr learned my addr 172.21.15.55:0/4202984378 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.495+0000 7effdf7fe640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fae30 0x7effe010c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effdeffd640 1 -- 172.21.15.55:0/4202984378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fae30 msgr2=0x7effe010c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effdeffd640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fae30 0x7effe010c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effdeffd640 1 -- 172.21.15.55:0/4202984378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd00098b0 con 0x7effe00fc050 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effdeffd640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7effd4002a50 tx=0x7effd4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd4007c30 con 0x7effe00fc050 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effd400f040 con 0x7effe00fc050 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0111f10 con 0x7effe00fc050 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effd400daf0 con 0x7effe00fc050 2023-12-19T19:12:21.500 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.496+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe0112480 con 0x7effe00fc050 2023-12-19T19:12:21.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.497+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe01038a0 con 0x7effe00fc050 2023-12-19T19:12:21.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.497+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7effd400dc90 con 0x7effe00fc050 2023-12-19T19:12:21.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.498+0000 7effc7fff640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 0x7effc806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.498+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7effd40865f0 con 0x7effe00fc050 2023-12-19T19:12:21.501 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.498+0000 7effdf7fe640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 0x7effc806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.502 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.498+0000 7effdf7fe640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 0x7effc806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7effd0003400 tx=0x7effd0037040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.504 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.502+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7effd404ca00 con 0x7effe00fc050 2023-12-19T19:12:21.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.677+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7effe0112b70 con 0x7effe00fc050 2023-12-19T19:12:21.679 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.677+0000 7effc7fff640 1 -- 172.21.15.55:0/4202984378 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7effd4052af0 con 0x7effe00fc050 2023-12-19T19:12:21.680 INFO:teuthology.orchestra.run.smithi055.stdout:pg_num: 32 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 msgr2=0x7effc806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 0x7effc806b380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7effd0003400 tx=0x7effd0037040 comp rx=0 tx=0).stop 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 msgr2=0x7effe010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7effd4002a50 tx=0x7effd4004600 comp rx=0 tx=0).stop 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 shutdown_connections 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7effc8068ed0 0x7effc806b380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7effe00fc050 0x7effe010ccd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 --2- 172.21.15.55:0/4202984378 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effe00fae30 0x7effe010c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 >> 172.21.15.55:0/4202984378 conn(0x7effe00f6580 msgr2=0x7effe0064130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 shutdown_connections 2023-12-19T19:12:21.682 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.679+0000 7effe5cae640 1 -- 172.21.15.55:0/4202984378 wait complete. 2023-12-19T19:12:21.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 -- 172.21.15.55:0/3439614774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fbfc0 msgr2=0x7fe9300fc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 --2- 172.21.15.55:0/3439614774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fbfc0 0x7fe9300fc430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe9240048d0 tx=0x7fe924009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 -- 172.21.15.55:0/3439614774 shutdown_connections 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 --2- 172.21.15.55:0/3439614774 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fbfc0 0x7fe9300fc430 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 --2- 172.21.15.55:0/3439614774 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fae30 0x7fe9300fb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.793+0000 7fe93679d640 1 -- 172.21.15.55:0/3439614774 >> 172.21.15.55:0/3439614774 conn(0x7fe9300f6510 msgr2=0x7fe9300f8980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 -- 172.21.15.55:0/3439614774 shutdown_connections 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 -- 172.21.15.55:0/3439614774 wait complete. 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 Processor -- start 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 -- start start 2023-12-19T19:12:21.800 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fbfc0 0x7fe9301114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930111a10 con 0x7fe9300fae30 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe93679d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe930111b50 con 0x7fe9300fbfc0 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56230/0 (socket says 172.21.15.55:56230) 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.794+0000 7fe92ffff640 1 -- 172.21.15.55:0/431615363 learned_addr learned my addr 172.21.15.55:0/431615363 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92f7fe640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fbfc0 0x7fe9301114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92ffff640 1 -- 172.21.15.55:0/431615363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fbfc0 msgr2=0x7fe9301114d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92ffff640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fbfc0 0x7fe9301114d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92ffff640 1 -- 172.21.15.55:0/431615363 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe924005040 con 0x7fe9300fae30 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92ffff640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe920002a80 tx=0x7fe920004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe920007c30 con 0x7fe9300fae30 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe92000f040 con 0x7fe9300fae30 2023-12-19T19:12:21.801 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9200095b0 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe930111e30 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.795+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe930112270 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.796+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe930103830 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.797+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe920009710 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.797+0000 7fe92cff9640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 0x7fe91806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.797+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe9200866e0 con 0x7fe9300fae30 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.797+0000 7fe92f7fe640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 0x7fe91806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:21.802 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.797+0000 7fe92f7fe640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 0x7fe91806b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe9300649d0 tx=0x7fe924009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:21.803 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:21.801+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe92004ca00 con 0x7fe9300fae30 2023-12-19T19:12:22.011 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.009+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9301124e0 con 0x7fe9300fae30 2023-12-19T19:12:22.012 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.009+0000 7fe92cff9640 1 -- 172.21.15.55:0/431615363 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+4683 (secure 0 0 0) 0x7fe920052be0 con 0x7fe9300fae30 2023-12-19T19:12:22.012 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-19T19:12:22.012 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi055.qvunsw","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.55:6829/58770560","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6828","nonce":58770560},{"type":"v1","addr":"172.21.15.55:6829","nonce":58770560}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24267,"name":"cephfs.smithi187.uciyzu","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.187:6825/191360506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":191360506},{"type":"v1","addr":"172.21.15.187:6825","nonce":191360506}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24281,"name":"cephfs.smithi187.xdgtcm","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.187:6827/1691713828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":1691713828},{"type":"v1","addr":"172.21.15.187:6827","nonce":1691713828}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-19T19:11:51.982744+0000","modified":"2023-12-19T19:12:08.014106+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi055.wupdfj","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.55:6827/3287430109","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":3287430109},{"type":"v1","addr":"172.21.15.55:6827","nonce":3287430109}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-19T19:12:22.012 INFO:teuthology.orchestra.run.smithi055.stderr:dumped fsmap epoch 12 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 msgr2=0x7fe91806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 0x7fe91806b380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe9300649d0 tx=0x7fe924009da0 comp rx=0 tx=0).stop 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 msgr2=0x7fe930064c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe920002a80 tx=0x7fe920004600 comp rx=0 tx=0).stop 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 shutdown_connections 2023-12-19T19:12:22.014 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9300fbfc0 0x7fe9301114d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:22.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe918068ed0 0x7fe91806b380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:22.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 --2- 172.21.15.55:0/431615363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe9300fae30 0x7fe930064c20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:22.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.011+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 >> 172.21.15.55:0/431615363 conn(0x7fe9300f6510 msgr2=0x7fe9300ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:22.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.012+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 shutdown_connections 2023-12-19T19:12:22.015 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:22.012+0000 7fe93679d640 1 -- 172.21.15.55:0/431615363 wait complete. 2023-12-19T19:12:22.025 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 12, 'max_mds': 1, 'flags': 18} 2023-12-19T19:12:22.025 INFO:teuthology.run_tasks:Running task kclient... 2023-12-19T19:12:22.055 INFO:tasks.kclient:Mounting kernel clients... 2023-12-19T19:12:22.056 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-12-19T19:12:22.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:22.056 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2023-12-19T19:12:22.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:22.074 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link delete ceph-brx 2023-12-19T19:12:22.147 INFO:teuthology.orchestra.run.smithi055.stderr:Cannot find device "ceph-brx" 2023-12-19T19:12:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T19:12:22.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:22.148 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2023-12-19T19:12:22.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:22.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2023-12-19T19:12:22.237 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:21 smithi187 ceph-mon[87698]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:22.237 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:21 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/868687044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.237 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:21 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/3181996565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.237 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:21 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/4202984378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.245 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2023-12-19T19:12:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T19:12:22.246 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-12-19T19:12:22.246 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-19T19:12:22.246 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:22.247 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-19T19:12:22.274 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:21 smithi055 ceph-mon[80978]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:22.274 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:21 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/868687044' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.274 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:21 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/3181996565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.275 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:21 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/4202984378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-19T19:12:22.515 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:23.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:22 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/431615363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T19:12:23.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:22 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/431615363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-19T19:12:24.070 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:23 smithi055 ceph-mon[80978]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:24.075 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.072+0000 7f78935fc700 1 -- 172.21.15.55:0/3606962468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 msgr2=0x7f788c101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:24.075 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.072+0000 7f78935fc700 1 --2- 172.21.15.55:0/3606962468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c101ed0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7888004940 tx=0x7f7888009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 -- 172.21.15.55:0/3606962468 shutdown_connections 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 --2- 172.21.15.55:0/3606962468 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 0x7f788c0fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 --2- 172.21.15.55:0/3606962468 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c101ed0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 -- 172.21.15.55:0/3606962468 >> 172.21.15.55:0/3606962468 conn(0x7f788c0719b0 msgr2=0x7f788c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 -- 172.21.15.55:0/3606962468 shutdown_connections 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 -- 172.21.15.55:0/3606962468 wait complete. 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 Processor -- start 2023-12-19T19:12:24.076 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.073+0000 7f78935fc700 1 -- start start 2023-12-19T19:12:24.077 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f78935fc700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 0x7f788c10e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f78935fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f78935fc700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c10f1a0 con 0x7f788c101b00 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f78935fc700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c10f310 con 0x7f788c0fbb40 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:56244/0 (socket says 172.21.15.55:56244) 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 -- 172.21.15.55:0/1527437179 learned_addr learned my addr 172.21.15.55:0/1527437179 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7891398700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 0x7f788c10e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 -- 172.21.15.55:0/1527437179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 msgr2=0x7f788c10e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 0x7f788c10e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 -- 172.21.15.55:0/1527437179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888005040 con 0x7f788c101b00 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7890b97700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c8c0 tx=0x7f787c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787c00a980 con 0x7f788c101b00 2023-12-19T19:12:24.078 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f787c00ce60 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c1176a0 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.074+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f787c0113a0 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.075+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c117b60 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.076+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c103fb0 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.076+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f787c012400 con 0x7f788c101b00 2023-12-19T19:12:24.079 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.076+0000 7f7881ffb700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 0x7f787806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:24.080 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.077+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f787c086ec0 con 0x7f788c101b00 2023-12-19T19:12:24.080 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.077+0000 7f7891398700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 0x7f787806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:24.080 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.077+0000 7f7891398700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 0x7f787806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7888004f90 tx=0x7f7888009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:24.083 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.081+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f787c053390 con 0x7f788c101b00 2023-12-19T19:12:24.324 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.321+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f788c0734d0 con 0x7f788c101b00 2023-12-19T19:12:24.324 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.321+0000 7f7881ffb700 1 -- 172.21.15.55:0/1527437179 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+83 (secure 0 0 0) 0x7f787c0569b0 con 0x7f788c101b00 2023-12-19T19:12:24.324 INFO:teuthology.orchestra.run.smithi055.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 msgr2=0x7f787806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 0x7f787806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7888004f90 tx=0x7f7888009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 msgr2=0x7f788c10ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c8c0 tx=0x7f787c0044c0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 shutdown_connections 2023-12-19T19:12:24.326 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f7878068f60 0x7f787806b410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f788c0fbb40 0x7f788c10e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 --2- 172.21.15.55:0/1527437179 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f788c101b00 0x7f788c10ec10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:24.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.323+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 >> 172.21.15.55:0/1527437179 conn(0x7f788c0719b0 msgr2=0x7f788c0fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:24.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.324+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 shutdown_connections 2023-12-19T19:12:24.327 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:24.324+0000 7f78935fc700 1 -- 172.21.15.55:0/1527437179 wait complete. 2023-12-19T19:12:24.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:23 smithi187 ceph-mon[87698]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi055.front.sepia.ceph.com 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T19:12:24.973 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T19:12:24.974 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T19:12:24.974 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-19T19:12:24.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:24.974 DEBUG:teuthology.orchestra.run.smithi055:> ip addr 2023-12-19T19:12:25.031 INFO:teuthology.orchestra.run.smithi055.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T19:12:25.031 INFO:teuthology.orchestra.run.smithi055.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T19:12:25.031 INFO:teuthology.orchestra.run.smithi055.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T19:12:25.031 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 ::1/128 scope host 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d6 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f0 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether 0c:c4:7a:6b:fd:d7 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp7s0f1 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:06 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f0 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: inet 172.21.15.55/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-19T19:12:25.032 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft 42129sec preferred_lft 42129sec 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run.smithi055.stdout: inet6 fe80::ae1f:6bff:fef8:2606/64 scope link noprefixroute 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run.smithi055.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run.smithi055.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run.smithi055.stdout: link/ether ac:1f:6b:f8:26:07 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run.smithi055.stdout: altname enp1s0f1 2023-12-19T19:12:25.033 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T19:12:25.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add name ceph-brx type bridge 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr flush dev ceph-brx 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set ceph-brx up 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T19:12:25.034 DEBUG:teuthology.orchestra.run.smithi055:> ') 2023-12-19T19:12:25.200 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:24 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1527437179' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T19:12:25.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.222 DEBUG:teuthology.orchestra.run.smithi055:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T19:12:25.252 INFO:teuthology.orchestra.run.smithi055.stdout:1 2023-12-19T19:12:25.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.254 DEBUG:teuthology.orchestra.run.smithi055:> route 2023-12-19T19:12:25.332 INFO:teuthology.orchestra.run.smithi055.stdout:Kernel IP routing table 2023-12-19T19:12:25.332 INFO:teuthology.orchestra.run.smithi055.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T19:12:25.333 INFO:teuthology.orchestra.run.smithi055.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-19T19:12:25.333 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-19T19:12:25.333 INFO:teuthology.orchestra.run.smithi055.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T19:12:25.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-19T19:12:25.339 DEBUG:teuthology.orchestra.run.smithi055:> ') 2023-12-19T19:12:25.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:24 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1527437179' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T19:12:25.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.618 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list 2023-12-19T19:12:25.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.634 DEBUG:teuthology.orchestra.run.smithi055:> ip netns list-id 2023-12-19T19:12:25.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.690 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2023-12-19T19:12:25.690 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2023-12-19T19:12:25.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-19T19:12:25.690 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-19T19:12:25.690 DEBUG:teuthology.orchestra.run.smithi055:> ') 2023-12-19T19:12:25.780 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-19T19:12:25.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-19T19:12:25.780 DEBUG:teuthology.orchestra.run.smithi055:> ') 2023-12-19T19:12:25.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:25.898 DEBUG:teuthology.orchestra.run.smithi055:> (cd / && exec bash -c ' 2023-12-19T19:12:25.898 DEBUG:teuthology.orchestra.run.smithi055:> set -e 2023-12-19T19:12:25.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set brx.0 up 2023-12-19T19:12:25.898 DEBUG:teuthology.orchestra.run.smithi055:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T19:12:25.898 DEBUG:teuthology.orchestra.run.smithi055:> ') 2023-12-19T19:12:25.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T19:12:25.949 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:25.964 INFO:teuthology.orchestra.run.smithi055.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-19T19:12:25.964 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T19:12:25.964 DEBUG:teuthology.orchestra.run.smithi055:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:26.019 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-19T19:12:26.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.020 DEBUG:teuthology.orchestra.run.smithi055:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T19:12:26.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:25 smithi055 ceph-mon[80978]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:26.101 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T19:12:26.101 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T19:12:26.101 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2023-12-19T19:12:26.220 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T19:12:26.220 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T19:12:26.220 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2023-12-19T19:12:26.220 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.187:3300" 2023-12-19T19:12:26.220 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T19:12:26.225 INFO:teuthology.orchestra.run.smithi055.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:invalid new device string format 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.187:3300" 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: resolved to: "172.21.15.55:6789,172.21.15.187:6789" 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.55:6789,172.21.15.187:6789:/ 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=63de9ba6-9ea1-11ee-95a5-87774f69a715" will pass to kernel 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout: You just mounted a file system that supports labels which does not 2023-12-19T19:12:26.226 INFO:teuthology.orchestra.run.smithi055.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-19T19:12:26.227 INFO:teuthology.orchestra.run.smithi055.stdout: applications will generate AVC messages and not be allowed access to 2023-12-19T19:12:26.227 INFO:teuthology.orchestra.run.smithi055.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-19T19:12:26.228 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-19T19:12:26.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.228 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:26.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.260 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:26.324 INFO:teuthology.orchestra.run.smithi055.stdout:client14524 2023-12-19T19:12:26.325 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:12:26.325 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:12:26.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:25 smithi187 ceph-mon[87698]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:26.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.382 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:26.444 INFO:teuthology.orchestra.run.smithi055.stdout:client14524 2023-12-19T19:12:26.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.444 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:26.506 INFO:teuthology.orchestra.run.smithi055.stdout:client14524 2023-12-19T19:12:26.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:26.506 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715.client14524/status') 2023-12-19T19:12:26.580 INFO:teuthology.orchestra.run.smithi055.stdout:instance: client.14524 (3)192.168.0.1:0/3895560104 2023-12-19T19:12:26.580 INFO:teuthology.orchestra.run.smithi055.stdout:blocklisted: false 2023-12-19T19:12:26.580 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records in 2023-12-19T19:12:26.581 INFO:teuthology.orchestra.run.smithi055.stderr:0+1 records out 2023-12-19T19:12:26.581 INFO:teuthology.orchestra.run.smithi055.stderr:70 bytes copied, 2.1601e-05 s, 3.2 MB/s 2023-12-19T19:12:26.582 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-12-19T19:12:26.582 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-19T19:12:26.582 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:26.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-19T19:12:26.894 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:28.281 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:27 smithi055 ceph-mon[80978]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:28.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:27 smithi187 ceph-mon[87698]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:28.508 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- 172.21.15.55:0/1375197249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fdc70 msgr2=0x7f93b40fe040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 --2- 172.21.15.55:0/1375197249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fdc70 0x7f93b40fe040 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f93a4004940 tx=0x7f93a4009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- 172.21.15.55:0/1375197249 shutdown_connections 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 --2- 172.21.15.55:0/1375197249 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fe610 0x7f93b4106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 --2- 172.21.15.55:0/1375197249 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fdc70 0x7f93b40fe040 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- 172.21.15.55:0/1375197249 >> 172.21.15.55:0/1375197249 conn(0x7f93b40f76c0 msgr2=0x7f93b40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- 172.21.15.55:0/1375197249 shutdown_connections 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- 172.21.15.55:0/1375197249 wait complete. 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 Processor -- start 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.504+0000 7f93bc251700 1 -- start start 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fdc70 0x7f93b41136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4114290 con 0x7f93b40fe610 2023-12-19T19:12:28.509 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b410d580 con 0x7f93b40fdc70 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37024/0 (socket says 172.21.15.55:37024) 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 -- 172.21.15.55:0/1362238493 learned_addr learned my addr 172.21.15.55:0/1362238493 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b9fed700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fdc70 0x7f93b41136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 -- 172.21.15.55:0/1362238493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fdc70 msgr2=0x7f93b41136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fdc70 0x7f93b41136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 -- 172.21.15.55:0/1362238493 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4005040 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93b97ec700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f93b0002a50 tx=0x7f93b00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b0007cc0 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b410d810 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93b000f040 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93b000d9c0 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.505+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b410dd50 con 0x7f93b40fe610 2023-12-19T19:12:28.510 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.506+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b4104260 con 0x7f93b40fe610 2023-12-19T19:12:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.507+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f93b000db20 con 0x7f93b40fe610 2023-12-19T19:12:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.507+0000 7f93aa7fc700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 0x7f93a006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.507+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f93b0086ad0 con 0x7f93b40fe610 2023-12-19T19:12:28.511 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.507+0000 7f93b9fed700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 0x7f93a006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:28.512 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.508+0000 7f93b9fed700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 0x7f93a006b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009fd0 tx=0x7f93a4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:28.514 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.512+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f93b00569b0 con 0x7f93b40fe610 2023-12-19T19:12:28.761 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.758+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f93b404bb80 con 0x7f93b40fe610 2023-12-19T19:12:28.761 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.758+0000 7f93aa7fc700 1 -- 172.21.15.55:0/1362238493 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+83 (secure 0 0 0) 0x7f93b0056540 con 0x7f93b40fe610 2023-12-19T19:12:28.761 INFO:teuthology.orchestra.run.smithi055.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-19T19:12:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 msgr2=0x7f93a006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 0x7f93a006b410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009fd0 tx=0x7f93a4009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 msgr2=0x7f93b4113c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f93b0002a50 tx=0x7f93b00046b0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.763 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 shutdown_connections 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f93a0068f60 0x7f93a006b410 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93b40fdc70 0x7f93b41136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 --2- 172.21.15.55:0/1362238493 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93b40fe610 0x7f93b4113c40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.760+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 >> 172.21.15.55:0/1362238493 conn(0x7f93b40f76c0 msgr2=0x7f93b41013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.761+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 shutdown_connections 2023-12-19T19:12:28.764 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:28.761+0000 7f93bc251700 1 -- 172.21.15.55:0/1362238493 wait complete. 2023-12-19T19:12:29.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:29 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1362238493' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T19:12:29.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:29 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1362238493' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-19T19:12:29.515 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-19T19:12:29.515 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-19T19:12:29.515 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-19T19:12:29.516 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-19T19:12:29.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:29.516 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:29.536 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f0 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f1 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42136sec preferred_lft 42136sec 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec4:7aff:fe88:77b1/64 scope link 2023-12-19T19:12:29.537 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2023-12-19T19:12:29.537 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-19T19:12:29.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add name ceph-brx type bridge 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr flush dev ceph-brx 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set ceph-brx up 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-19T19:12:29.538 DEBUG:teuthology.orchestra.run.smithi187:> ') 2023-12-19T19:12:29.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:29.740 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-19T19:12:29.768 INFO:teuthology.orchestra.run.smithi187.stdout:1 2023-12-19T19:12:29.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:29.770 DEBUG:teuthology.orchestra.run.smithi187:> route 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-19T19:12:29.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-19T19:12:29.840 DEBUG:teuthology.orchestra.run.smithi187:> ') 2023-12-19T19:12:30.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.109 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2023-12-19T19:12:30.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.124 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2023-12-19T19:12:30.167 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:30 smithi187 ceph-mon[87698]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:30.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.179 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2023-12-19T19:12:30.179 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2023-12-19T19:12:30.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-19T19:12:30.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-19T19:12:30.180 DEBUG:teuthology.orchestra.run.smithi187:> ') 2023-12-19T19:12:30.264 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-19T19:12:30.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.264 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-19T19:12:30.265 DEBUG:teuthology.orchestra.run.smithi187:> ') 2023-12-19T19:12:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:30 smithi055 ceph-mon[80978]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:30.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.365 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec bash -c ' 2023-12-19T19:12:30.365 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2023-12-19T19:12:30.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2023-12-19T19:12:30.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.0 master ceph-brx 2023-12-19T19:12:30.365 DEBUG:teuthology.orchestra.run.smithi187:> ') 2023-12-19T19:12:30.407 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T19:12:30.407 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:30.462 INFO:teuthology.orchestra.run.smithi187.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-19T19:12:30.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-19T19:12:30.462 DEBUG:teuthology.orchestra.run.smithi187:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:30.516 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-19T19:12:30.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T19:12:30.619 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T19:12:30.619 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T19:12:30.619 INFO:teuthology.orchestra.run.smithi187.stdout:invalid new device string format 2023-12-19T19:12:30.736 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T19:12:30.736 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T19:12:30.736 INFO:teuthology.orchestra.run.smithi187.stdout:invalid new device string format 2023-12-19T19:12:30.737 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.187:3300" 2023-12-19T19:12:30.737 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T19:12:30.742 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-19T19:12:30.742 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-19T19:12:30.742 INFO:teuthology.orchestra.run.smithi187.stdout:invalid new device string format 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.55:3300,172.21.15.187:3300" 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.55:6789,172.21.15.187:6789" 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.55:6789,172.21.15.187:6789:/ 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=63de9ba6-9ea1-11ee-95a5-87774f69a715" will pass to kernel 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout: You just mounted a file system that supports labels which does not 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout: applications will generate AVC messages and not be allowed access to 2023-12-19T19:12:30.743 INFO:teuthology.orchestra.run.smithi187.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-19T19:12:30.745 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-19T19:12:30.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:30.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.772 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:30.828 INFO:teuthology.orchestra.run.smithi187.stdout:client14530 2023-12-19T19:12:30.828 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:12:30.828 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:12:30.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.884 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:30.940 INFO:teuthology.orchestra.run.smithi187.stdout:client14530 2023-12-19T19:12:30.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.941 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-19T19:12:30.997 INFO:teuthology.orchestra.run.smithi187.stdout:client14530 2023-12-19T19:12:30.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:12:30.998 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715.client14530/status') 2023-12-19T19:12:31.064 INFO:teuthology.orchestra.run.smithi187.stdout:instance: client.14530 (3)172.21.15.187:0/1182275014 2023-12-19T19:12:31.064 INFO:teuthology.orchestra.run.smithi187.stdout:blocklisted: false 2023-12-19T19:12:31.065 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2023-12-19T19:12:31.065 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2023-12-19T19:12:31.065 INFO:teuthology.orchestra.run.smithi187.stderr:72 bytes copied, 2.2745e-05 s, 3.2 MB/s 2023-12-19T19:12:31.066 INFO:teuthology.run_tasks:Running task print... 2023-12-19T19:12:31.077 INFO:teuthology.task.print:**** done client 2023-12-19T19:12:31.078 INFO:teuthology.run_tasks:Running task parallel... 2023-12-19T19:12:31.092 INFO:teuthology.task.parallel:starting parallel... 2023-12-19T19:12:31.093 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-19T19:12:31.094 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-19T19:12:31.094 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:12:31.094 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-12-19T19:12:31.095 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-19T19:12:31.095 INFO:teuthology.task.sequential:In sequential, running task workunit... 2023-12-19T19:12:31.100 INFO:tasks.workunit:Pulling workunits from ref c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:12:31.101 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-19T19:12:31.101 INFO:tasks.workunit:timeout=3h 2023-12-19T19:12:31.101 INFO:tasks.workunit:cleanup=True 2023-12-19T19:12:31.101 DEBUG:teuthology.orchestra.run.smithi055:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 94h/148d Inode: 1 Links: 2 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-19 19:11:56.026202939 +0000 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-19 19:12:26.256776708 +0000 2023-12-19T19:12:31.122 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: 2023-12-19 19:11:56.026202939 +0000 2023-12-19T19:12:31.123 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-19T19:12:31.123 DEBUG:teuthology.orchestra.run.smithi055:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-19T19:12:31.201 DEBUG:teuthology.orchestra.run.smithi187:> stat -- /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 73h/115d Inode: 1 Links: 3 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-19T19:12:31.218 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-19T19:12:31.219 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2023-12-19 19:12:31.193680595 +0000 2023-12-19T19:12:31.219 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2023-12-19 19:12:31.193680595 +0000 2023-12-19T19:12:31.219 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2023-12-19 19:11:56.026202939 +0000 2023-12-19T19:12:31.219 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-12-19T19:12:31.219 DEBUG:teuthology.orchestra.run.smithi187:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-12-19T19:12:31.289 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:12:31.290 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:12:31.306 INFO:tasks.workunit.client.0.smithi055.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-19T19:12:31.344 INFO:tasks.workunit.client.1.smithi187.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-12-19T19:12:31.385 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:32.164 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:32 smithi055 ceph-mon[80978]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:32.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:32 smithi187 ceph-mon[87698]: pgmap v197: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2023-12-19T19:12:32.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- 172.21.15.55:0/2086536363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 msgr2=0x7f93a00fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:32.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 --2- 172.21.15.55:0/2086536363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a00fe640 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9390004940 tx=0x7f9390009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:32.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- 172.21.15.55:0/2086536363 shutdown_connections 2023-12-19T19:12:32.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 --2- 172.21.15.55:0/2086536363 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 0x7f93a00facc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:32.944 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 --2- 172.21.15.55:0/2086536363 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a00fe640 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- 172.21.15.55:0/2086536363 >> 172.21.15.55:0/2086536363 conn(0x7f93a00f69d0 msgr2=0x7f93a00f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- 172.21.15.55:0/2086536363 shutdown_connections 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- 172.21.15.55:0/2086536363 wait complete. 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 Processor -- start 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.941+0000 7f93a5e41700 1 -- start start 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f93a5e41700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f93a5e41700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 0x7f93a010dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f93a5e41700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a010e380 con 0x7f93a00fe230 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f93a5e41700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a010ec20 con 0x7f93a00feb80 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37042/0 (socket says 172.21.15.55:37042) 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 -- 172.21.15.55:0/1458016150 learned_addr learned my addr 172.21.15.55:0/1458016150 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939effd700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 0x7f93a010dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 -- 172.21.15.55:0/1458016150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 msgr2=0x7f93a010dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:32.945 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 0x7f93a010dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:32.946 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 -- 172.21.15.55:0/1458016150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9390005040 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f939f7fe700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9390009380 tx=0x7f9390009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939000e040 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f939000bd20 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.942+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a010eea0 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.943+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939001c630 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.943+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a010f3e0 con 0x7f93a00fe230 2023-12-19T19:12:32.947 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.944+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a01075e0 con 0x7f93a00fe230 2023-12-19T19:12:32.948 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.944+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9390013420 con 0x7f93a00fe230 2023-12-19T19:12:32.948 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.944+0000 7f9387fff700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 0x7f938806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:32.948 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.945+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f93900881d0 con 0x7f93a00fe230 2023-12-19T19:12:32.948 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.945+0000 7f939effd700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 0x7f938806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:32.948 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.945+0000 7f939effd700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 0x7f938806b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9394004f40 tx=0x7f939400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:32.951 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:32.949+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9390058090 con 0x7f93a00fe230 2023-12-19T19:12:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.147+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f93a004bb80 con 0x7f93a00fe230 2023-12-19T19:12:33.150 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.147+0000 7f9387fff700 1 -- 172.21.15.55:0/1458016150 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f9390057c20 con 0x7f93a00fe230 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 msgr2=0x7f938806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 0x7f938806b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9394004f40 tx=0x7f939400b040 comp rx=0 tx=0).stop 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 msgr2=0x7f93a010d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9390009380 tx=0x7f9390009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 shutdown_connections 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f9388068f60 0x7f938806b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f93a00fe230 0x7f93a010d840 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 --2- 172.21.15.55:0/1458016150 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93a00feb80 0x7f93a010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 >> 172.21.15.55:0/1458016150 conn(0x7f93a00f69d0 msgr2=0x7f93a01013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:33.152 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 shutdown_connections 2023-12-19T19:12:33.153 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:33.149+0000 7f93a5e41700 1 -- 172.21.15.55:0/1458016150 wait complete. 2023-12-19T19:12:33.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:33 smithi055 ceph-mon[80978]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T19:12:33.858 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-12-19T19:12:34.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:33 smithi187 ceph-mon[87698]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T19:12:34.169 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:35.248 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:35.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.860+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/2160117833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 msgr2=0x7fb9f00fd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.860+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/2160117833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f00fd530 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0004940 tx=0x7fb9e0009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/2160117833 shutdown_connections 2023-12-19T19:12:35.865 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/2160117833 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 0x7fb9f00fc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/2160117833 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f00fd530 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/2160117833 >> 172.21.15.55:0/2160117833 conn(0x7fb9f00f6950 msgr2=0x7fb9f00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/2160117833 shutdown_connections 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/2160117833 wait complete. 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 Processor -- start 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- start start 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 0x7fb9f01102e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00fbb20 con 0x7fb9f00fd120 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.861+0000 7fb9f5ac3700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00fbc60 con 0x7fb9f00fbe30 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:35.866 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:37066/0 (socket says 172.21.15.55:37066) 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 -- 172.21.15.55:0/3542424708 learned_addr learned my addr 172.21.15.55:0/3542424708 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9ef7fe700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 0x7fb9f01102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 -- 172.21.15.55:0/3542424708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 msgr2=0x7fb9f01102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 0x7fb9f01102e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 -- 172.21.15.55:0/3542424708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e0005040 con 0x7fb9f00fd120 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9eeffd700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4002a80 tx=0x7fb9e40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e4007cc0 con 0x7fb9f00fd120 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb9e400f040 con 0x7fb9f00fd120 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f0112e90 con 0x7fb9f00fd120 2023-12-19T19:12:35.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e400d9a0 con 0x7fb9f00fd120 2023-12-19T19:12:35.868 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.862+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f01133a0 con 0x7fb9f00fd120 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.863+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0106510 con 0x7fb9f00fd120 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.864+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb9e400db00 con 0x7fb9f00fd120 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.864+0000 7fb9d7fff700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 0x7fb9d806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.864+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb9e4086ab0 con 0x7fb9f00fd120 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.864+0000 7fb9ef7fe700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 0x7fb9d806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:35.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.865+0000 7fb9ef7fe700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 0x7fb9d806b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e0009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:35.871 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:35.869+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb9e4052f00 con 0x7fb9f00fd120 2023-12-19T19:12:36.069 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.067+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7fb9f01044d0 con 0x7fb9f00fd120 2023-12-19T19:12:36.070 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.067+0000 7fb9d7fff700 1 -- 172.21.15.55:0/3542424708 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb9e4056520 con 0x7fb9f00fd120 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 msgr2=0x7fb9d806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 0x7fb9d806b420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb9e0000c00 tx=0x7fb9e0009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 msgr2=0x7fb9f0112830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4002a80 tx=0x7fb9e40046b0 comp rx=0 tx=0).stop 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 shutdown_connections 2023-12-19T19:12:36.072 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb9d8068f70 0x7fb9d806b420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:36.073 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9f00fbe30 0x7fb9f01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:36.073 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 --2- 172.21.15.55:0/3542424708 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb9f00fd120 0x7fb9f0112830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:36.073 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.069+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 >> 172.21.15.55:0/3542424708 conn(0x7fb9f00f6950 msgr2=0x7fb9f00f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:36.073 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.070+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 shutdown_connections 2023-12-19T19:12:36.073 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:36.070+0000 7fb9f5ac3700 1 -- 172.21.15.55:0/3542424708 wait complete. 2023-12-19T19:12:36.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:35 smithi055 ceph-mon[80978]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T19:12:36.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:35 smithi187 ceph-mon[87698]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2023-12-19T19:12:36.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set global log_to_journald false --force' 2023-12-19T19:12:36.994 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:38.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:37 smithi055 ceph-mon[80978]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T19:12:38.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:37 smithi187 ceph-mon[87698]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T19:12:38.672 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- 172.21.15.55:0/4031632758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fcbf0 msgr2=0x7f50600fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:38.672 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 --2- 172.21.15.55:0/4031632758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fcbf0 0x7f50600fd000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5054004940 tx=0x7f5054009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:38.672 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- 172.21.15.55:0/4031632758 shutdown_connections 2023-12-19T19:12:38.672 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 --2- 172.21.15.55:0/4031632758 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fde00 0x7f50600fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 --2- 172.21.15.55:0/4031632758 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fcbf0 0x7f50600fd000 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- 172.21.15.55:0/4031632758 >> 172.21.15.55:0/4031632758 conn(0x7f50600f82d0 msgr2=0x7f50600fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- 172.21.15.55:0/4031632758 shutdown_connections 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- 172.21.15.55:0/4031632758 wait complete. 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 Processor -- start 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.669+0000 7f50670aa700 1 -- start start 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f50670aa700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 0x7f5060071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f50670aa700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f50670aa700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060070590 con 0x7f50600fde00 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f50670aa700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5060070700 con 0x7f50600fcbf0 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f5064e46700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 0x7f5060071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:38.673 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35206/0 (socket says 172.21.15.55:35206) 2023-12-19T19:12:38.674 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 -- 172.21.15.55:0/1126542439 learned_addr learned my addr 172.21.15.55:0/1126542439 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 -- 172.21.15.55:0/1126542439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 msgr2=0x7f5060071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 0x7f5060071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f505ffff700 1 -- 172.21.15.55:0/1126542439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5054005040 con 0x7f50600fde00 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.670+0000 7f5064e46700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 0x7f5060071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f505ffff700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f504c002a50 tx=0x7f504c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504c007cc0 con 0x7f50600fde00 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50600709e0 con 0x7f50600fde00 2023-12-19T19:12:38.675 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f504c00f040 con 0x7f50600fde00 2023-12-19T19:12:38.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504c00d9c0 con 0x7f50600fde00 2023-12-19T19:12:38.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.671+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5060115590 con 0x7f50600fde00 2023-12-19T19:12:38.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.672+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50601075d0 con 0x7f50600fde00 2023-12-19T19:12:38.676 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.673+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f504c00db40 con 0x7f50600fde00 2023-12-19T19:12:38.677 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.673+0000 7f505d7fa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 0x7f505006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:38.677 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.673+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f504c086ac0 con 0x7f50600fde00 2023-12-19T19:12:38.677 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.673+0000 7f5064e46700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 0x7f505006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:38.677 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.674+0000 7f5064e46700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 0x7f505006b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f505400b7d0 tx=0x7f5054009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:38.684 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.681+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f504c056a50 con 0x7f50600fde00 2023-12-19T19:12:38.882 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.880+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f506004bb80 con 0x7f50600fde00 2023-12-19T19:12:38.882 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.880+0000 7f505d7fa700 1 -- 172.21.15.55:0/1126542439 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f506004bb80 con 0x7f50600fde00 2023-12-19T19:12:38.884 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 msgr2=0x7f505006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 0x7f505006b410 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f505400b7d0 tx=0x7f5054009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 msgr2=0x7f506006ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f504c002a50 tx=0x7f504c0046b0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 shutdown_connections 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f5050068f60 0x7f505006b410 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50600fcbf0 0x7f5060071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 --2- 172.21.15.55:0/1126542439 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f50600fde00 0x7f506006ff90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 >> 172.21.15.55:0/1126542439 conn(0x7f50600f82d0 msgr2=0x7f50600fa590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.882+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 shutdown_connections 2023-12-19T19:12:38.885 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:38.883+0000 7f50670aa700 1 -- 172.21.15.55:0/1126542439 wait complete. 2023-12-19T19:12:39.548 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2023-12-19T19:12:39.827 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:12:40.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:39 smithi187 ceph-mon[87698]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T19:12:40.283 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:39 smithi055 ceph-mon[80978]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2023-12-19T19:12:41.610 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- 172.21.15.55:0/186530381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 msgr2=0x7fe2a80fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/186530381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a80fe250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe294004940 tx=0x7fe294009e30 comp rx=0 tx=0).stop 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- 172.21.15.55:0/186530381 shutdown_connections 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/186530381 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a80fe250 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/186530381 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 0x7fe2a80fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- 172.21.15.55:0/186530381 >> 172.21.15.55:0/186530381 conn(0x7fe2a80f82d0 msgr2=0x7fe2a80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- 172.21.15.55:0/186530381 shutdown_connections 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- 172.21.15.55:0/186530381 wait complete. 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 Processor -- start 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.607+0000 7fe2aebc4700 1 -- start start 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2aebc4700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 0x7fe2a810d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2aebc4700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2aebc4700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a810dfd0 con 0x7fe2a80fde00 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2aebc4700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a810e870 con 0x7fe2a80fcbf0 2023-12-19T19:12:41.611 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:35218/0 (socket says 172.21.15.55:35218) 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 -- 172.21.15.55:0/2316306109 learned_addr learned my addr 172.21.15.55:0/2316306109 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 -- 172.21.15.55:0/2316306109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 msgr2=0x7fe2a810d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2ac960700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 0x7fe2a810d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 0x7fe2a810d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:12:41.612 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.608+0000 7fe2a7fff700 1 -- 172.21.15.55:0/2316306109 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294005040 con 0x7fe2a80fde00 2023-12-19T19:12:41.613 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2a7fff700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe294008f90 tx=0x7fe294009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:41.613 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe294021070 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe29401c400 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a810eaf0 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe29400e030 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.609+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a810f030 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.610+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a804bb80 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.611+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe29401ca00 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.611+0000 7fe2a57fa700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 0x7fe29806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.611+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe294088160 con 0x7fe2a80fde00 2023-12-19T19:12:41.614 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.611+0000 7fe2ac960700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 0x7fe29806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:12:41.615 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.611+0000 7fe2ac960700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 0x7fe29806b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe29c004600 tx=0x7fe29c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:12:41.618 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.615+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe2940580c0 con 0x7fe2a80fde00 2023-12-19T19:12:41.856 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:12:41.853+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2a810f4f0 con 0x7fe298068f70 2023-12-19T19:12:42.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:41 smithi055 ceph-mon[80978]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:12:42.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:41 smithi187 ceph-mon[87698]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:12:43.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:42 smithi187 ceph-mon[87698]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:12:43.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:42 smithi055 ceph-mon[80978]: from='client.14546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:12:44.305 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:43 smithi055 ceph-mon[80978]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:12:44.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:43 smithi187 ceph-mon[87698]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:12:46.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:45 smithi055 ceph-mon[80978]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:12:46.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:45 smithi187 ceph-mon[87698]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:12:48.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:47 smithi055 ceph-mon[80978]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:12:48.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:47 smithi187 ceph-mon[87698]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:12:50.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:49 smithi055 ceph-mon[80978]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:50.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:49 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:49 smithi187 ceph-mon[87698]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:50.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:49 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:12:52.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:51 smithi055 ceph-mon[80978]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:52.314 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:51 smithi187 ceph-mon[87698]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:54.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:53 smithi055 ceph-mon[80978]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:54.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:53 smithi187 ceph-mon[87698]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:56.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:55 smithi055 ceph-mon[80978]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:56.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:55 smithi187 ceph-mon[87698]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:58.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:57 smithi055 ceph-mon[80978]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:12:58.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:57 smithi187 ceph-mon[87698]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:00.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:12:59 smithi055 ceph-mon[80978]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:00.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:12:59 smithi187 ceph-mon[87698]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:02.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:01 smithi055 ceph-mon[80978]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:02.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:01 smithi187 ceph-mon[87698]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:04.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:03 smithi055 ceph-mon[80978]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:04.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:03 smithi187 ceph-mon[87698]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:05.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:04 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:05.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:04 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:06.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:05 smithi055 ceph-mon[80978]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:06.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:05 smithi187 ceph-mon[87698]: pgmap v214: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-19T19:13:08.088 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:07 smithi055 ceph-mon[80978]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:13:08.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:07 smithi187 ceph-mon[87698]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2023-12-19T19:13:08.866 INFO:teuthology.orchestra.run.smithi055.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:13:08.867 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.863+0000 7fe2a57fa700 1 -- 172.21.15.55:0/2316306109 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fe29c01a070 con 0x7fe298068f70 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 msgr2=0x7fe29806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 0x7fe29806b420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe29c004600 tx=0x7fe29c00b040 comp rx=0 tx=0).stop 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 msgr2=0x7fe2a810d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe294008f90 tx=0x7fe294009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 shutdown_connections 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fe298068f70 0x7fe29806b420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2a80fcbf0 0x7fe2a810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 --2- 172.21.15.55:0/2316306109 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fe2a80fde00 0x7fe2a810d9d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:08.869 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.866+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 >> 172.21.15.55:0/2316306109 conn(0x7fe2a80f82d0 msgr2=0x7fe2a8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:08.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.867+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 shutdown_connections 2023-12-19T19:13:08.870 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:08.867+0000 7fe2aebc4700 1 -- 172.21.15.55:0/2316306109 wait complete. 2023-12-19T19:13:09.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2023-12-19T19:13:09.966 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:13:10.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:09 smithi055 ceph-mon[80978]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:13:10.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:10.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:09 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:13:10.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:09 smithi055 ceph-mon[80978]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:09 smithi187 ceph-mon[87698]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:13:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:09 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:13:10.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:09 smithi187 ceph-mon[87698]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:11.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:13:11.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:13:11.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:10 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:11.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:10 smithi055 ceph-mon[80978]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:13:11.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-19T19:13:11.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-19T19:13:11.241 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:10 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:11.242 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:10 smithi187 ceph-mon[87698]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-19T19:13:12.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:11 smithi055 ceph-mon[80978]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:12.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:11 smithi187 ceph-mon[87698]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:13.803 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:13 smithi055 ceph-mon[80978]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:14.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:13 smithi187 ceph-mon[87698]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:16.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:15 smithi055 ceph-mon[80978]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:16.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:15 smithi187 ceph-mon[87698]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:16.559 INFO:tasks.workunit.client.1.smithi187.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-19T19:13:16.559 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.559 INFO:tasks.workunit.client.1.smithi187.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:state without impacting any branches by switching back to a branch. 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: git switch -c 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:Or undo this operation with: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: git switch - 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr: 2023-12-19T19:13:16.560 INFO:tasks.workunit.client.1.smithi187.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-19T19:13:16.566 DEBUG:teuthology.orchestra.run.smithi187:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-12-19T19:13:16.630 INFO:tasks.workunit.client.1.smithi187.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-19T19:13:16.632 INFO:tasks.workunit.client.1.smithi187.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-19T19:13:16.632 INFO:tasks.workunit.client.1.smithi187.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-19T19:13:17.417 INFO:tasks.workunit.client.1.smithi187.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-19T19:13:17.485 INFO:tasks.workunit.client.1.smithi187.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-19T19:13:17.528 INFO:tasks.workunit.client.1.smithi187.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-19T19:13:17.528 INFO:tasks.workunit.client.1.smithi187.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-19T19:13:17.528 INFO:tasks.workunit.client.1.smithi187.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-19T19:13:17.566 INFO:tasks.workunit.client.1.smithi187.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-19T19:13:17.569 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:13:17.569 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-12-19T19:13:17.625 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2023-12-19T19:13:17.626 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-19T19:13:17.626 DEBUG:teuthology.orchestra.run.smithi187:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2023-12-19T19:13:17.686 INFO:tasks.workunit.client.1.smithi187.stderr:+ mkdir -p fsstress 2023-12-19T19:13:17.688 INFO:tasks.workunit.client.1.smithi187.stderr:+ pushd fsstress 2023-12-19T19:13:17.688 INFO:tasks.workunit.client.1.smithi187.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-19T19:13:17.688 INFO:tasks.workunit.client.1.smithi187.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 -- 172.21.15.55:0/1608119612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 msgr2=0x7f66300fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 --2- 172.21.15.55:0/1608119612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f66300fe250 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6618004940 tx=0x7f6618009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 -- 172.21.15.55:0/1608119612 shutdown_connections 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 --2- 172.21.15.55:0/1608119612 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f66300fe250 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 --2- 172.21.15.55:0/1608119612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 0x7f66300fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 -- 172.21.15.55:0/1608119612 >> 172.21.15.55:0/1608119612 conn(0x7f66300f82d0 msgr2=0x7f66300fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 -- 172.21.15.55:0/1608119612 shutdown_connections 2023-12-19T19:13:18.117 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.431+0000 7f6634f77700 1 -- 172.21.15.55:0/1608119612 wait complete. 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 Processor -- start 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 -- start start 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 0x7f6630071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6630070590 con 0x7f66300fde00 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f6634f77700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6630070700 con 0x7f66300fcbf0 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:49322/0 (socket says 172.21.15.55:49322) 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 -- 172.21.15.55:0/519161212 learned_addr learned my addr 172.21.15.55:0/519161212 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662e59c700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 0x7f6630071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 -- 172.21.15.55:0/519161212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 msgr2=0x7f6630071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 0x7f6630071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.118 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.432+0000 7f662dd9b700 1 -- 172.21.15.55:0/519161212 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618005040 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f662dd9b700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6618009380 tx=0x7f6618009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661800e040 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630070980 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f661800bd20 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f661801c630 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.433+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6630115540 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.434+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663004bb80 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.435+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6618013420 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.435+0000 7f6626ffd700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 0x7f661c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.435+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6618088100 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.435+0000 7f662e59c700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 0x7f661c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.436+0000 7f662e59c700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 0x7f661c06b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66300718a0 tx=0x7f6620009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.440+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6618058060 con 0x7f66300fde00 2023-12-19T19:13:18.119 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.664+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6630102750 con 0x7f661c068f60 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.665+0000 7f6626ffd700 1 -- 172.21.15.55:0/519161212 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f6630102750 con 0x7f661c068f60 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 msgr2=0x7f661c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 0x7f661c06b410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66300718a0 tx=0x7f6620009da0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 msgr2=0x7f663006ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6618009380 tx=0x7f6618009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 shutdown_connections 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f661c068f60 0x7f661c06b410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f66300fcbf0 0x7f6630071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 --2- 172.21.15.55:0/519161212 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f66300fde00 0x7f663006ff90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 >> 172.21.15.55:0/519161212 conn(0x7f66300f82d0 msgr2=0x7f6630101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.120 INFO:teuthology.orchestra.run.smithi055.stdout:true 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 shutdown_connections 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.667+0000 7f6634f77700 1 -- 172.21.15.55:0/519161212 wait complete. 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 -- 172.21.15.55:0/3639023182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 msgr2=0x7f92300fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 --2- 172.21.15.55:0/3639023182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f92300fb2e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9220004940 tx=0x7f9220009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 -- 172.21.15.55:0/3639023182 shutdown_connections 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 --2- 172.21.15.55:0/3639023182 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 0x7f92300fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 --2- 172.21.15.55:0/3639023182 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f92300fb2e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.777+0000 7f923618c700 1 -- 172.21.15.55:0/3639023182 >> 172.21.15.55:0/3639023182 conn(0x7f92300f6950 msgr2=0x7f92300f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 -- 172.21.15.55:0/3639023182 shutdown_connections 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 -- 172.21.15.55:0/3639023182 wait complete. 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 Processor -- start 2023-12-19T19:13:18.122 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 -- start start 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 0x7f923010db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923010e120 con 0x7f92300faed0 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f923618c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923010e9c0 con 0x7f92300fdf60 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.778+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41944/0 (socket says 172.21.15.55:41944) 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 -- 172.21.15.55:0/712600635 learned_addr learned my addr 172.21.15.55:0/712600635 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f9226ffd700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 0x7f923010db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 -- 172.21.15.55:0/712600635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 msgr2=0x7f923010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 0x7f923010db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 -- 172.21.15.55:0/712600635 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9220005040 con 0x7f92300faed0 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922f7fe700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9220009fd0 tx=0x7f9220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.123 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922000e040 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923010ec40 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9220013420 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f922001c630 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.779+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923010f180 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.780+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92301075f0 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.781+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9220013a00 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.781+0000 7f922cff9700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 0x7f921c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.781+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f9220088050 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.782+0000 7f9226ffd700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 0x7f921c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.782+0000 7f9226ffd700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 0x7f921c06b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9218004f40 tx=0x7f921800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:17.787+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9220057fb0 con 0x7f92300faed0 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.004+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f923002b180 con 0x7f921c068f60 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.005+0000 7f922cff9700 1 -- 172.21.15.55:0/712600635 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f923002b180 con 0x7f921c068f60 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 msgr2=0x7f921c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.124 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 0x7f921c06b410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9218004f40 tx=0x7f921800b040 comp rx=0 tx=0).stop 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 msgr2=0x7f923010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9220009fd0 tx=0x7f9220009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 shutdown_connections 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f921c068f60 0x7f921c06b410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f92300faed0 0x7f923010d5e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 --2- 172.21.15.55:0/712600635 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92300fdf60 0x7f923010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 >> 172.21.15.55:0/712600635 conn(0x7f92300f6950 msgr2=0x7f92300f8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.125 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 shutdown_connections 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.007+0000 7f923618c700 1 -- 172.21.15.55:0/712600635 wait complete. 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/3972285384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 msgr2=0x7fb5f80fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/3972285384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f80fd150 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec004940 tx=0x7fb5ec009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/3972285384 shutdown_connections 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/3972285384 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f80fd150 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/3972285384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 0x7fb5f80fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/3972285384 >> 172.21.15.55:0/3972285384 conn(0x7fb5f80f71d0 msgr2=0x7fb5f80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/3972285384 shutdown_connections 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.112+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/3972285384 wait complete. 2023-12-19T19:13:18.126 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 Processor -- start 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 -- start start 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 0x7fb5f810d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f810e030 con 0x7fb5f80fcd00 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.113+0000 7fb5fd7b6700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f810e8d0 con 0x7fb5f80fbaf0 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41966/0 (socket says 172.21.15.55:41966) 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 -- 172.21.15.55:0/4067998368 learned_addr learned my addr 172.21.15.55:0/4067998368 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f6ffd700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 0x7fb5f810d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 -- 172.21.15.55:0/4067998368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 msgr2=0x7fb5f810d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 0x7fb5f810d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 -- 172.21.15.55:0/4067998368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec005040 con 0x7fb5f80fcd00 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5f67fc700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009380 tx=0x7fb5ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ec00e040 con 0x7fb5f80fcd00 2023-12-19T19:13:18.127 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5ec00bd20 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.114+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ec01c630 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.115+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f810eb50 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.115+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f810f090 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.116+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f804bb80 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.116+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb5ec013420 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.117+0000 7fb5df7fe700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 0x7fb5e006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.117+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb5ec088c20 con 0x7fb5f80fcd00 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.117+0000 7fb5f6ffd700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 0x7fb5e006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.128 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.117+0000 7fb5f6ffd700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 0x7fb5e006b4e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80fcb60 tx=0x7fb5e8009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.129 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.121+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb5ec058b60 con 0x7fb5f80fcd00 2023-12-19T19:13:18.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:17 smithi055 ceph-mon[80978]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:13:18.333 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.331+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5f810f540 con 0x7fb5e0069030 2023-12-19T19:13:18.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:17 smithi187 ceph-mon[87698]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 597 B/s wr, 0 op/s 2023-12-19T19:13:18.345 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.342+0000 7fb5df7fe700 1 -- 172.21.15.55:0/4067998368 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fb5f810f540 con 0x7fb5e0069030 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:alertmanager.smithi055 smithi055 *:9093,9094 running (5m) 58s ago 6m 24.7M - 0.25.0 c8568f914cd2 526d8c1dbdb9 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi055 smithi055 running (7m) 58s ago 7m 6665k - 17.2.7-131-ga88f3edf f9ecdbaa1030 53b90cb7fb26 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:crash.smithi187 smithi187 running (6m) 64s ago 6m 6660k - 17.2.7-131-ga88f3edf f9ecdbaa1030 52d1b402aa76 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:grafana.smithi055 smithi055 *:3000 running (5m) 58s ago 6m 83.7M - 9.4.7 954c08fa6188 c4acfbe0e286 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.qvunsw smithi055 running (77s) 58s ago 77s 13.5M - 17.2.7-131-ga88f3edf f9ecdbaa1030 e29bb285f9fc 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi055.wupdfj smithi055 running (83s) 58s ago 83s 13.5M - 17.2.7-131-ga88f3edf f9ecdbaa1030 25e6ba18c009 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi187.uciyzu smithi187 running (81s) 64s ago 80s 12.0M - 17.2.7-131-ga88f3edf f9ecdbaa1030 85b35b278b69 2023-12-19T19:13:18.346 INFO:teuthology.orchestra.run.smithi055.stdout:mds.cephfs.smithi187.xdgtcm smithi187 running (74s) 64s ago 74s 11.9M - 17.2.7-131-ga88f3edf f9ecdbaa1030 3336afaafe4c 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi055.dwclvp smithi055 *:9283 running (8m) 58s ago 8m 479M - 17.2.7-131-ga88f3edf f9ecdbaa1030 c99d4ccfe879 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:mgr.smithi187.pmhvpw smithi187 *:8443,9283 running (5m) 64s ago 5m 432M - 17.2.7-131-ga88f3edf f9ecdbaa1030 36e60396eb8d 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi055 smithi055 running (8m) 58s ago 8m 46.8M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 22d858bf8f44 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:mon.smithi187 smithi187 running (5m) 64s ago 5m 38.7M 2048M 17.2.7-131-ga88f3edf f9ecdbaa1030 31c58c5214bc 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi055 smithi055 *:9100 running (6m) 58s ago 6m 15.6M - 1.5.0 0da6a335fe13 4477ddc2ccc2 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:node-exporter.smithi187 smithi187 *:9100 running (5m) 64s ago 5m 16.5M - 1.5.0 0da6a335fe13 8a84ed919750 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.0 smithi055 running (5m) 58s ago 5m 45.4M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 b82fbcd42bf0 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.1 smithi055 running (4m) 58s ago 4m 48.0M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 0e674f452739 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.2 smithi055 running (4m) 58s ago 4m 46.2M 4289M 17.2.7-131-ga88f3edf f9ecdbaa1030 338fa21e9e0b 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.3 smithi187 running (3m) 64s ago 3m 44.1M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 153484b5fa97 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.4 smithi187 running (3m) 64s ago 3m 41.5M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 96c21caf7e77 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:osd.5 smithi187 running (2m) 64s ago 2m 41.0M 5312M 17.2.7-131-ga88f3edf f9ecdbaa1030 7c6c250a7b10 2023-12-19T19:13:18.347 INFO:teuthology.orchestra.run.smithi055.stdout:prometheus.smithi055 smithi055 *:9095 running (5m) 58s ago 6m 56.6M - 2.43.0 a07b618ecd1d a51dcacc4149 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 msgr2=0x7fb5e006b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 0x7fb5e006b4e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80fcb60 tx=0x7fb5e8009040 comp rx=0 tx=0).stop 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 msgr2=0x7fb5f810da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec009380 tx=0x7fb5ec009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 shutdown_connections 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7fb5e0069030 0x7fb5e006b4e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb5f80fbaf0 0x7fb5f810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 --2- 172.21.15.55:0/4067998368 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7fb5f80fcd00 0x7fb5f810da30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 >> 172.21.15.55:0/4067998368 conn(0x7fb5f80f71d0 msgr2=0x7fb5f8065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.348 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 shutdown_connections 2023-12-19T19:13:18.349 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.344+0000 7fb5fd7b6700 1 -- 172.21.15.55:0/4067998368 wait complete. 2023-12-19T19:13:18.452 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- 172.21.15.55:0/3374484968 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 msgr2=0x7f2ee40ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.452 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/3374484968 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee40ff800 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8004940 tx=0x7f2ed8009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.452 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- 172.21.15.55:0/3374484968 shutdown_connections 2023-12-19T19:13:18.453 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/3374484968 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 0x7f2ee41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.453 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/3374484968 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee40ff800 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.453 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- 172.21.15.55:0/3374484968 >> 172.21.15.55:0/3374484968 conn(0x7f2ee40f7190 msgr2=0x7f2ee40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.453 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- 172.21.15.55:0/3374484968 shutdown_connections 2023-12-19T19:13:18.453 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- 172.21.15.55:0/3374484968 wait complete. 2023-12-19T19:13:18.454 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 Processor -- start 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.450+0000 7f2eeb01e700 1 -- start start 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2eeb01e700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2eeb01e700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 0x7f2ee410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2eeb01e700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee410e000 con 0x7f2ee4066030 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2eeb01e700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee410e8a0 con 0x7f2ee40ffdd0 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:41974/0 (socket says 172.21.15.55:41974) 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 -- 172.21.15.55:0/1919096264 learned_addr learned my addr 172.21.15.55:0/1919096264 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee3fff700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 0x7f2ee410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 -- 172.21.15.55:0/1919096264 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 msgr2=0x7f2ee410da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 0x7f2ee410da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 -- 172.21.15.55:0/1919096264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed8005040 con 0x7f2ee4066030 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.451+0000 7f2ee8dba700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.455 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.452+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed8021070 con 0x7f2ee4066030 2023-12-19T19:13:18.456 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.452+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ed801c400 con 0x7f2ee4066030 2023-12-19T19:13:18.456 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.452+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee410eb20 con 0x7f2ee4066030 2023-12-19T19:13:18.456 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.452+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ed800e070 con 0x7f2ee4066030 2023-12-19T19:13:18.456 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.452+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee410f000 con 0x7f2ee4066030 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.453+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee404bb80 con 0x7f2ee4066030 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.453+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2ed8013a30 con 0x7f2ee4066030 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.454+0000 7f2ee17fa700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 0x7f2ed406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.454+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2ed8087f90 con 0x7f2ee4066030 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.454+0000 7f2ee3fff700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 0x7f2ed406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.457 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.454+0000 7f2ee3fff700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 0x7f2ed406b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009b80 tx=0x7f2ed000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.460 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.458+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2ed8057ef0 con 0x7f2ee4066030 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.737+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2ee410f6c0 con 0x7f2ee4066030 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.737+0000 7f2ee17fa700 1 -- 172.21.15.55:0/1919096264 <== mon.0 v2:172.21.15.55:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f2ed8057a80 con 0x7f2ee4066030 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: "mon": { 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: "mgr": { 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 2 2023-12-19T19:13:18.740 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "osd": { 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 6 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "mds": { 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 4 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: }, 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "overall": { 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: "ceph version 17.2.7-131-ga88f3edf (a88f3edf230e1c58be7443afb566f4b63b8b9c94) quincy (stable)": 14 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout: } 2023-12-19T19:13:18.741 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-19T19:13:18.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.739+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 msgr2=0x7f2ed406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.739+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 0x7f2ed406b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009b80 tx=0x7f2ed000b040 comp rx=0 tx=0).stop 2023-12-19T19:13:18.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.739+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 msgr2=0x7f2ee410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.739+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed8009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.742 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 shutdown_connections 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7f2ed4068f60 0x7f2ed406b410 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7f2ee4066030 0x7f2ee410d4c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 --2- 172.21.15.55:0/1919096264 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2ee40ffdd0 0x7f2ee410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 >> 172.21.15.55:0/1919096264 conn(0x7f2ee40f7190 msgr2=0x7f2ee40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 shutdown_connections 2023-12-19T19:13:18.743 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.740+0000 7f2eeb01e700 1 -- 172.21.15.55:0/1919096264 wait complete. 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 -- 172.21.15.55:0/406001712 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 msgr2=0x7ff50c0fbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 --2- 172.21.15.55:0/406001712 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c0fbf10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc004940 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 -- 172.21.15.55:0/406001712 shutdown_connections 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 --2- 172.21.15.55:0/406001712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 0x7ff50c0fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 --2- 172.21.15.55:0/406001712 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c0fbf10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 -- 172.21.15.55:0/406001712 >> 172.21.15.55:0/406001712 conn(0x7ff50c0f71c0 msgr2=0x7ff50c0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 -- 172.21.15.55:0/406001712 shutdown_connections 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.846+0000 7ff51169c700 1 -- 172.21.15.55:0/406001712 wait complete. 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 Processor -- start 2023-12-19T19:13:18.849 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 -- start start 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 0x7ff50c1127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 -- --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0669a0 con 0x7ff50c0fbb00 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff51169c700 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c066ae0 con 0x7ff50c0fcd10 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 --2- >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.55:3300/0 says I am v2:172.21.15.55:42006/0 (socket says 172.21.15.55:42006) 2023-12-19T19:13:18.851 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 -- 172.21.15.55:0/253956776 learned_addr learned my addr 172.21.15.55:0/253956776 (peer_addr_for_me v2:172.21.15.55:0/0) 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50a7fc700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 0x7ff50c1127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 -- 172.21.15.55:0/253956776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 msgr2=0x7ff50c1127e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 0x7ff50c1127e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 -- 172.21.15.55:0/253956776 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc005040 con 0x7ff50c0fbb00 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.847+0000 7ff50affd700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc004f90 tx=0x7ff4fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.848+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc00e040 con 0x7ff50c0fbb00 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.848+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4fc00bc50 con 0x7ff50c0fbb00 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.848+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c112de0 con 0x7ff50c0fbb00 2023-12-19T19:13:18.852 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.848+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff4fc01c630 con 0x7ff50c0fbb00 2023-12-19T19:13:18.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.848+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c113240 con 0x7ff50c0fbb00 2023-12-19T19:13:18.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.849+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 --> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c1064e0 con 0x7ff50c0fbb00 2023-12-19T19:13:18.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.849+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff4fc013420 con 0x7ff50c0fbb00 2023-12-19T19:13:18.853 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.850+0000 7ff4f37fe700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 0x7ff4f406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-19T19:13:18.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.850+0000 7ff50a7fc700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 0x7ff4f406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-19T19:13:18.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.850+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff4fc088150 con 0x7ff50c0fbb00 2023-12-19T19:13:18.854 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.850+0000 7ff50a7fc700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 0x7ff4f406b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff500004f40 tx=0x7ff50000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-19T19:13:18.856 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:18.854+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mon.0 v2:172.21.15.55:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff4fc054620 con 0x7ff50c0fbb00 2023-12-19T19:13:18.960 INFO:tasks.workunit.client.1.smithi187.stderr:+ tar xzf ltp-full.tgz 2023-12-19T19:13:19.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:18 smithi055 ceph-mon[80978]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:19.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:18 smithi055 ceph-mon[80978]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:19.053 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:18 smithi055 ceph-mon[80978]: from='client.? 172.21.15.55:0/1919096264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T19:13:19.093 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.090+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 --> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff50c02b170 con 0x7ff4f4068f60 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.092+0000 7ff4f37fe700 1 -- 172.21.15.55:0/253956776 <== mgr.14208 v2:172.21.15.55:6800/2515201024 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7ff50c02b170 con 0x7ff4f4068f60 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stdout: "in_progress": true, 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stdout: "which": "Upgrading daemons of type(s) mgr", 2023-12-19T19:13:19.094 INFO:teuthology.orchestra.run.smithi055.stdout: "services_complete": [], 2023-12-19T19:13:19.095 INFO:teuthology.orchestra.run.smithi055.stdout: "progress": "", 2023-12-19T19:13:19.095 INFO:teuthology.orchestra.run.smithi055.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 image", 2023-12-19T19:13:19.095 INFO:teuthology.orchestra.run.smithi055.stdout: "is_paused": false 2023-12-19T19:13:19.095 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 msgr2=0x7ff4f406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 0x7ff4f406b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff500004f40 tx=0x7ff50000b040 comp rx=0 tx=0).stop 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 msgr2=0x7ff50c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc004f90 tx=0x7ff4fc009e30 comp rx=0 tx=0).stop 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 shutdown_connections 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:6800/2515201024,v1:172.21.15.55:6801/2515201024] conn(0x7ff4f4068f60 0x7ff4f406b410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] conn(0x7ff50c0fbb00 0x7ff50c110290 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 --2- 172.21.15.55:0/253956776 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff50c0fcd10 0x7ff50c1127e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 >> 172.21.15.55:0/253956776 conn(0x7ff50c0f71c0 msgr2=0x7ff50c0f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 shutdown_connections 2023-12-19T19:13:19.097 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:13:19.094+0000 7ff51169c700 1 -- 172.21.15.55:0/253956776 wait complete. 2023-12-19T19:13:19.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:18 smithi187 ceph-mon[87698]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:19.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:18 smithi187 ceph-mon[87698]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:19.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:18 smithi187 ceph-mon[87698]: from='client.? 172.21.15.55:0/1919096264' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-19T19:13:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:19 smithi055 ceph-mon[80978]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:19 smithi055 ceph-mon[80978]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:20.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:19 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:19 smithi187 ceph-mon[87698]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:19 smithi187 ceph-mon[87698]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2023-12-19T19:13:20.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:19 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:21.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:20 smithi055 ceph-mon[80978]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:21.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:20 smithi187 ceph-mon[87698]: from='client.14566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-19T19:13:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:21 smithi055 ceph-mon[80978]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s wr, 0 op/s 2023-12-19T19:13:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:13:22.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:21 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:21 smithi187 ceph-mon[87698]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.1 KiB/s wr, 0 op/s 2023-12-19T19:13:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-19T19:13:22.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:21 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' 2023-12-19T19:13:23.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:22 smithi055 ceph-mon[80978]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-19T19:13:23.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:22 smithi055 ceph-mon[80978]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-19T19:13:23.304 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:22 smithi055 ceph-mon[80978]: Upgrade: Need to upgrade myself (mgr.smithi055.dwclvp) 2023-12-19T19:13:23.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:22 smithi187 ceph-mon[87698]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-19T19:13:23.341 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:22 smithi187 ceph-mon[87698]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-19T19:13:23.342 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:22 smithi187 ceph-mon[87698]: Upgrade: Need to upgrade myself (mgr.smithi055.dwclvp) 2023-12-19T19:13:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:23 smithi055 ceph-mon[80978]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi187 2023-12-19T19:13:24.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:23 smithi055 ceph-mon[80978]: pgmap v223: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 348 KiB/s wr, 1 op/s 2023-12-19T19:13:24.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:23 smithi187 ceph-mon[87698]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi187 2023-12-19T19:13:24.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:23 smithi187 ceph-mon[87698]: pgmap v223: 65 pgs: 65 active+clean; 4.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 348 KiB/s wr, 1 op/s 2023-12-19T19:13:26.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:25 smithi055 ceph-mon[80978]: pgmap v224: 65 pgs: 65 active+clean; 9.4 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 808 KiB/s wr, 84 op/s 2023-12-19T19:13:26.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:25 smithi187 ceph-mon[87698]: pgmap v224: 65 pgs: 65 active+clean; 9.4 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 808 KiB/s wr, 84 op/s 2023-12-19T19:13:28.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:28 smithi055 ceph-mon[80978]: pgmap v225: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 177 op/s 2023-12-19T19:13:28.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:28 smithi187 ceph-mon[87698]: pgmap v225: 65 pgs: 65 active+clean; 26 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 177 op/s 2023-12-19T19:13:30.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:30 smithi055 ceph-mon[80978]: pgmap v226: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 356 op/s 2023-12-19T19:13:30.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:30 smithi187 ceph-mon[87698]: pgmap v226: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 356 op/s 2023-12-19T19:13:32.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:32 smithi055 ceph-mon[80978]: pgmap v227: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 357 op/s 2023-12-19T19:13:32.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:32 smithi187 ceph-mon[87698]: pgmap v227: 65 pgs: 65 active+clean; 52 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.5 MiB/s wr, 357 op/s 2023-12-19T19:13:34.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:33 smithi055 ceph-mon[80978]: pgmap v228: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 358 op/s 2023-12-19T19:13:34.090 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:33 smithi187 ceph-mon[87698]: pgmap v228: 65 pgs: 65 active+clean; 55 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.8 MiB/s wr, 358 op/s 2023-12-19T19:13:35.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:34 smithi055 ceph-mon[80978]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:35.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:34 smithi187 ceph-mon[87698]: from='mgr.14208 172.21.15.55:0/4287288940' entity='mgr.smithi055.dwclvp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-19T19:13:36.293 INFO:tasks.workunit.client.1.smithi187.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2023-12-19T19:13:36.294 INFO:tasks.workunit.client.1.smithi187.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-19T19:13:36.294 INFO:tasks.workunit.client.1.smithi187.stderr:+ make 2023-12-19T19:13:36.303 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:13:35 smithi055 ceph-mon[80978]: pgmap v229: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 363 op/s 2023-12-19T19:13:36.340 INFO:journalctl@ceph.mon.smithi187.smithi187.stdout:Dec 19 19:13:35 smithi187 ceph-mon[87698]: pgmap v229: 65 pgs: 65 active+clean; 59 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 4.8 MiB/s wr, 363 op/s 2023-12-19T19:13:36.526 INFO:tasks.workunit.client.1.smithi187.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2023-12-19T19:13:58.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:57.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:58.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:57.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:58.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:57.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:state without impacting any branches by switching back to a branch. 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr: git switch -c 2023-12-19T19:13:58.254 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr:Or undo this operation with: 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr: git switch - 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr: 2023-12-19T19:13:58.255 INFO:tasks.workunit.client.0.smithi055.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-19T19:13:58.259 DEBUG:teuthology.orchestra.run.smithi055:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-19T19:13:58.285 INFO:tasks.workunit.client.0.smithi055.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-19T19:13:58.287 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-19T19:13:58.287 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-19T19:13:59.111 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-19T19:13:59.200 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-19T19:13:59.242 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-19T19:13:59.243 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-19T19:13:59.244 INFO:tasks.workunit.client.0.smithi055.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-19T19:13:59.284 INFO:tasks.workunit.client.0.smithi055.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-19T19:13:59.287 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:13:59.287 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-19T19:13:59.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:58.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:59.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:58.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:59.303 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:58.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:13:59.318 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2023-12-19T19:13:59.320 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-19T19:13:59.320 DEBUG:teuthology.orchestra.run.smithi055:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2023-12-19T19:14:00.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:59.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:00.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:59.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:00.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:13:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:13:59.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:00.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:00.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:00.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:00.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:00.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:00.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:01.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:00.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:01.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:00.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:01.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:00.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:01.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:00.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:01.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:00.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:01.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:00.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:01.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:01.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:01.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:01.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:01.557 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:01.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:02.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:01.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:02.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:01.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:02.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:01.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:02.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:01.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:02.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:01.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:02.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:01.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:02.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:02.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:02.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:02.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:02.660 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:02.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:02.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:02.766+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:02.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:02.766+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:02.964 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:02.766+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:03.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:02.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:03.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:02.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:03.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:02.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:03.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:03.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:03.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:03.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:03.759 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:03.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:04.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:04.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:04.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:04.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:03.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:04.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:03.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:04.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:03.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:04.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:04.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:04.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:04.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:04.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:04.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:05.011 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:04.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:05.011 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:04.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:05.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:04.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:05.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:04.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:05.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:04.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:05.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:05.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:05.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:05.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:06.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:05.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:06.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:05.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:06.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:05.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:06.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:05.717+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:06.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:05.717+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:06.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:05.717+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:06.761 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:06.376+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:06.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:06.376+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:06.762 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:06.376+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:07.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:06.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:07.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:06.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:07.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:06.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:07.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:06.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:07.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:06.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:07.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:06.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:07.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:07.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:07.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:07.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:07.654 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:07.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:08.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:07.724+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:08.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:07.724+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:08.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:07.724+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:08.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:08.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:08.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:08.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:08.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:08.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:08.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:08.314+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:08.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:08.314+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:08.678 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:08.314+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:08.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:08.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:08.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:09.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:09.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:09.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:09.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:09.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:09.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:09.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:09.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:09.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:09.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:09.677 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:09.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:10.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:09.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:10.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:09.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:10.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:09.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:10.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:10.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:10.304 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:10.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:10.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:10.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:10.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:10.699 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:10.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:10.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:10.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:10.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:10.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:10.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:10.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:11.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:11.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:11.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:10.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:11.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:11.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:11.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:11.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:11.681 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:11.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:12.024 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:11.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:12.024 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:11.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:12.024 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:11.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:12.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:12.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:12.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:12.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:12.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:12.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:12.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:12.364+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:12.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:12.364+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:12.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:12.364+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:13.052 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:12.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:13.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:12.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:13.053 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:12.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:13.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:13.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:13.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:13.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:13.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:13.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:13.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:13.591 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:13.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:14.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:13.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:14.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:13.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:14.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:13.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:14.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:14.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:14.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:14.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:14.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:14.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:14.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:14.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:14.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:14.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:14.640 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:14.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:14.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:14.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:14.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:15.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:15.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:15.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:15.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:15.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:15.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:15.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:15.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:15.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:15.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:15.626 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:15.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:16.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:15.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:15.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:15.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:16.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:16.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:16.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:16.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:16.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:16.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:16.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:16.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:16.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:16.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:16.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:16.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:17.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:17.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:17.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:17.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:17.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:17.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:17.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:17.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:17.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:17.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:17.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:17.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:17.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:18.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:18.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:18.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:18.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:18.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:18.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:18.605 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:18.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:18.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:18.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:18.606 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:18.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:19.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:18.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:18.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:18.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:19.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:19.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:19.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:19.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:19.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:19.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:19.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:19.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:19.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:20.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:20.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:20.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:20.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:20.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:20.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:20.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:20.585 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:20.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:20.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:20.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:20.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:20.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:20.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:20.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:21.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:20.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:21.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:20.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:21.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:20.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:21.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:21.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:21.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:21.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:21.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:21.804 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:21.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:22.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:22.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:22.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:22.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:22.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:22.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:22.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:22.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:22.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:22.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:22.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:22.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:22.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:23.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:23.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:23.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:23.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:23.374 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:23.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:23.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:23.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:23.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:23.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:23.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:23.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:23.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:24.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:24.069+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:24.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:24.069+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:24.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:24.069+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:24.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:24.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:24.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:24.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:24.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:24.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:24.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:25.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:25.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:25.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:25.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:25.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:25.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:25.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:25.318+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:25.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:25.318+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:25.805 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:25.318+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:25.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:25.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:25.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:25.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:25.806 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:25.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:26.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:26.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:26.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:26.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:26.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:26.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:26.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:26.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:26.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:26.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:26.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:26.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:26.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:27.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:27.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:27.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:27.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:27.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:27.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:27.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:27.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:27.666 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:27.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:27.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:28.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:28.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:28.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:28.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:28.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:28.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:29.005 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:28.596+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:29.005 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:28.596+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:29.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:28.596+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:29.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:29.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:29.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:29.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:29.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:29.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:29.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:30.012 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:29.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:30.012 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:29.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:30.012 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:29.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:30.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:30.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:30.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:30.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:30.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:30.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:30.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:30.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:30.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:30.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:30.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:31.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:30.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:31.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:30.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:31.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:30.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:31.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:31.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:31.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:31.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:31.284 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:31.004+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:31.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:31.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:31.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:31.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:31.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:31.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:32.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:31.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:32.028 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:31.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:32.028 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:31.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:32.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:32.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:32.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:32.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:32.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:32.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:32.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:33.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:32.592+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:33.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:32.592+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:33.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:32.592+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:33.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:33.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:33.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:33.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:33.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:33.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:33.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:33.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:33.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:33.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:33.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:33.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:33.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:33.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:33.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:33.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:33.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:33.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:34.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:34.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:34.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:34.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:34.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:34.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:34.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:34.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:34.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:34.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:34.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:34.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:34.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:35.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:35.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:35.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:35.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:35.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:35.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:35.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:35.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:35.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:35.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:35.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:35.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:35.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:35.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:35.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:35.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:35.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:35.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:36.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:36.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:36.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:36.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:36.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:36.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:36.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:36.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:36.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:36.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:36.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:36.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:36.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:36.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:36.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:37.274 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:37.275 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:37.275 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:36.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:37.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:37.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:37.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:37.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:37.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:37.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:37.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:38.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:37.930+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:38.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:37.930+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:38.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:37.930+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:38.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:38.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:38.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:38.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:38.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:38.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:38.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:38.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:38.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:38.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:38.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:38.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:38.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:38.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:38.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:39.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:39.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:39.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:39.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:39.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:39.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:39.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:39.298+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:39.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:39.298+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:39.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:39.298+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:40.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:39.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:40.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:39.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:40.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:39.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:40.623 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:40.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:40.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:40.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:40.624 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:40.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:40.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:40.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:40.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:40.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:40.624 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:40.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:40.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:40.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:40.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:41.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:41.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:41.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:41.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:41.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:41.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:41.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:41.308+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:41.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:41.308+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:41.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:41.308+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:42.268 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:41.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:42.268 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:41.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:42.268 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:41.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:42.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:42.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:42.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:42.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:42.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:42.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:42.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:42.912+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:42.912+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:43.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:42.912+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:43.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:43.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:43.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:43.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:43.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:43.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:43.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:43.288+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:43.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:43.288+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:43.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:43.288+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:44.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:43.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:44.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:43.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:44.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:43.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:44.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:44.334+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:44.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:44.334+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:44.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:44.334+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:44.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:44.469+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:44.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:44.469+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:44.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:44.469+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:45.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:44.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:45.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:44.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:45.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:44.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:45.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:45.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:45.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:45.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:45.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:45.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:45.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:45.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:45.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:45.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:45.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:45.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:46.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:45.854+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:46.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:45.854+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:46.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:45.854+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:46.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:46.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:46.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:46.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:46.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:46.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:46.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:46.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:46.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:46.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:46.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:46.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:47.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:46.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:47.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:46.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:47.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:46.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:47.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:47.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:47.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:47.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:47.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:47.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:47.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:47.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:47.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:47.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:47.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:47.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:48.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:47.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:48.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:47.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:48.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:47.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:48.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:48.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:48.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:48.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:48.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:48.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:48.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:48.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:48.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:48.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:48.876 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:48.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:49.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:48.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:49.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:48.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:49.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:48.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:49.513 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:49.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:49.514 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:49.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:49.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:49.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:49.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:49.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:49.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:50.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:49.846+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:50.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:49.846+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:50.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:49.846+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:50.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:50.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:50.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:50.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:50.464 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:50.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:50.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:50.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:50.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:50.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:50.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:50.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:51.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:50.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:51.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:50.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:51.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:50.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:51.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:51.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:51.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:51.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:51.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:51.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:51.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:51.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:51.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:51.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:51.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:51.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:52.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:51.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:52.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:51.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:52.233 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:51.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:52.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:52.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:52.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:52.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:52.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:52.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:52.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:52.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:52.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:52.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:52.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:52.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:53.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:52.830+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:53.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:52.830+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:53.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:52.830+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:53.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:53.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:53.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:53.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:53.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:53.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:53.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:53.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:53.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:53.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:53.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:53.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:54.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:53.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:54.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:53.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:54.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:53.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:54.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:54.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:54.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:54.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:54.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:54.472+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:54.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:54.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:54.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:54.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:54.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:54.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:55.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:54.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:55.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:54.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:55.255 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:54.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:55.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:55.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:55.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:55.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:55.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:55.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:55.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:55.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:55.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:55.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:55.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:55.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:56.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:55.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:56.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:55.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:56.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:55.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:56.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:56.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:56.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:56.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:56.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:56.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:56.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:56.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:56.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:56.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:56.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:56.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:57.282 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:56.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:57.282 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:56.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:57.282 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:56.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:57.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:57.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:57.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:57.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:57.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:57.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:57.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:58.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:57.898+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:58.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:57.898+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:58.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:57.898+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:58.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:58.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:58.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:58.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:58.295+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:58.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:58.295+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:58.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:58.295+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:59.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:58.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:59.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:58.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:59.296 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:58.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:59.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:59.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:14:59.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:59.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:59.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:14:59.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:14:59.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:00.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:59.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:00.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:59.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:00.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:14:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:14:59.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:00.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:00.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:00.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:00.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:00.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:00.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:00.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:01.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:00.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:01.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:00.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:01.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:00.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:01.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:01.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:01.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:01.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:01.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:01.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:01.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:01.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:01.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:01.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:01.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:01.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:02.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:01.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:02.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:01.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:02.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:01.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:02.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:02.309+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:02.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:02.309+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:02.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:02.309+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:02.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:02.285+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:02.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:02.285+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:02.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:02.285+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:03.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:03.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:03.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:03.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:03.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:03.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:03.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:03.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:03.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:03.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:04.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:04.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:04.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:04.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:04.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:04.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:04.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:04.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:04.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:04.350+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:04.350+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:04.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:04.350+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:05.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:05.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:05.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:05.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:05.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:05.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:05.325+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:05.325+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:05.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:05.325+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:06.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:06.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:06.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:05.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:06.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:06.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:06.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:06.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:06.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:06.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:06.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:07.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:07.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:07.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:07.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:07.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:07.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:07.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:07.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:08.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:07.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:08.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:07.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:08.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:07.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:07.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:07.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:08.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:08.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:08.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:08.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:08.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:08.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:08.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:08.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:08.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:08.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:08.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:08.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:09.311 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:09.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:09.311 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:09.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:09.311 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:09.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:09.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:09.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:09.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:09.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:09.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:09.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:09.802 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:09.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:09.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:09.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:09.803 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:09.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:10.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:10.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:10.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:10.303+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:10.303+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:10.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:10.303+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:10.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:10.097+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:10.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:10.097+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:10.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:10.097+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:11.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:11.294+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:11.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:11.294+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:11.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:11.294+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:11.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:11.101+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:11.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:11.101+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:11.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:11.101+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:11.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:11.258+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:11.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:11.258+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:11.554 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:11.258+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:12.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:12.283+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:12.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:12.283+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:12.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:12.283+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:12.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:12.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:12.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:12.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:12.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:12.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:12.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:12.129+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:12.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:12.129+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:12.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:12.129+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:13.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:13.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:13.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:13.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:13.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:13.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:13.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:13.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:13.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:13.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:13.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:13.516 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:13.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:14.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:14.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:14.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:14.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:14.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:14.304 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:14.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:14.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:14.047+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:14.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:14.047+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:14.305 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:14.047+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:15.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:15.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:15.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:15.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:15.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:15.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:15.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:15.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:15.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:15.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:15.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:15.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:16.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:16.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:16.247 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:15.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:16.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:16.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:16.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:16.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:16.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:16.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:16.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:16.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:16.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:16.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:16.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:17.266 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:16.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:17.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:16.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:17.267 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:16.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:17.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:17.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:17.301+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:17.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:17.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:17.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:17.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:18.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:17.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:18.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:17.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:18.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:17.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:18.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:18.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:18.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:18.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:18.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:18.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:18.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:19.253 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:18.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:19.253 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:18.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:19.253 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:18.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:19.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:19.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:19.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:19.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:19.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:19.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:19.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:19.339+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:19.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:19.339+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:19.553 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:19.339+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:20.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:19.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:20.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:19.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:20.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:19.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:20.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:20.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:20.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:20.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:20.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:20.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:20.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:21.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:20.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:21.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:20.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:21.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:20.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:21.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:21.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:21.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:21.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:21.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:21.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:21.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:21.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:21.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:21.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:21.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:21.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:22.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:21.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:22.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:21.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:22.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:21.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:22.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:22.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:22.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:22.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:22.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:22.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:22.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:23.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:22.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:23.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:22.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:23.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:22.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:23.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:23.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:23.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:23.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:23.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:23.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:23.263+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:24.253 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:23.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:24.254 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:23.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:24.254 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:23.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:24.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:24.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:24.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:24.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:24.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:24.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:24.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:25.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:24.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:25.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:24.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:25.209 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:24.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:25.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:25.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:25.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:25.208+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:25.208+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:25.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:25.208+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:26.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:25.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:26.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:25.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:26.218 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:25.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:26.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:26.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:26.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:26.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:26.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:26.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:26.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:26.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:26.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:26.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:26.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:26.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:27.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:26.879+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:27.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:26.879+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:27.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:26.879+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:27.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:27.197+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:27.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:27.197+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:27.463 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:27.197+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:27.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:27.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:27.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:28.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:27.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:28.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:27.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:28.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:27.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:28.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:28.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:28.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:28.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:28.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:28.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:28.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:28.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:28.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:28.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:28.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:28.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:29.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:28.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:29.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:28.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:29.217 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:28.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:29.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:29.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:29.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:29.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:29.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:29.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:29.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:29.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:29.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:29.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:29.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:29.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:30.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:29.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:30.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:29.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:30.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:29.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:30.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:30.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:30.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:30.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:30.531+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:30.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:30.531+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:30.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:30.531+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:31.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:30.834+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:31.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:30.834+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:31.241 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:30.834+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:31.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:31.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:31.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:31.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:31.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:31.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:31.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:31.505+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:31.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:31.505+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:31.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:31.505+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:31.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:31.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:32.222 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:31.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:32.478 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:32.221+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:32.478 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:32.221+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:32.478 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:32.221+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:32.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:32.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:32.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:32.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:32.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:32.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:33.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:32.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:33.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:32.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:33.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:32.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:33.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:33.251+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:33.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:33.251+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:33.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:33.251+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:33.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:33.520+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:33.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:33.520+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:33.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:33.520+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:34.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:33.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:34.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:33.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:34.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:33.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:34.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:34.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:34.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:34.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:34.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:34.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:34.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:34.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:34.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:34.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:34.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:34.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:35.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:34.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:35.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:34.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:35.053 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:34.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:35.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:35.458+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:35.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:35.458+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:35.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:35.458+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:35.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:35.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:35.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:35.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:35.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:35.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:36.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:35.724+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:36.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:35.724+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:36.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:35.724+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:36.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:36.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:36.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:36.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:36.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:36.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:36.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:36.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:36.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:36.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:36.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:36.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:37.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:36.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:37.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:36.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:37.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:36.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:37.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:37.244+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:37.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:37.244+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:37.429 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:37.244+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:37.686 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:37.429+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:37.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:37.429+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:37.687 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:37.429+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:38.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:37.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:38.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:37.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:38.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:37.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:38.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:38.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:38.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:38.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:38.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:38.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:38.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:38.261+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:38.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:38.261+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:38.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:38.261+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:39.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:38.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:39.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:38.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:39.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:38.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:39.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:39.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:39.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:39.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:39.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:39.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:39.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:39.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:39.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:39.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:39.553 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:39.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:40.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:39.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:40.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:39.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:40.052 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:39.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:40.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:40.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:40.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:40.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:40.552 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:40.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:40.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:40.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:40.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:40.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:40.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:40.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:41.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:40.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:41.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:40.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:41.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:40.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:41.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:41.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:41.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:41.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:41.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:41.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:41.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:41.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:41.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:41.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:41.552 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:41.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:42.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:41.755+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:42.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:41.755+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:42.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:41.755+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:42.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:42.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:42.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:42.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:42.454 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:42.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:42.776 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:15:42.661+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:15:42.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:42.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:42.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:42.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:42.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:42.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:43.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:42.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:43.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:42.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:43.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:42.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:43.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:43.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:43.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:43.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:43.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:43.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:43.754 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:43.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:43.754 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:43.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:43.754 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:43.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:44.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:43.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:44.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:43.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:44.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:43.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:44.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:44.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:44.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:44.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:44.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:44.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:44.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:44.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:44.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:44.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:44.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:44.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:44.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:44.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:44.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:44.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:44.802 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:44.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:45.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:45.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:45.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:45.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:45.500 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:45.180+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:45.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:45.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:45.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:45.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:45.783 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:45.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:46.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:45.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:46.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:45.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:46.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:45.783+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:46.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:46.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:46.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:46.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:46.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:46.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:46.488+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:46.488+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:46.488+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:47.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:46.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:47.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:46.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:47.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:46.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:47.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:47.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:47.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:47.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:47.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:47.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:47.781 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:15:47.661+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:15:47.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:47.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:47.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:47.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:47.782 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:47.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:48.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:47.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:48.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:47.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:48.054 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:47.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:48.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:48.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:48.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:48.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:48.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:48.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:48.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:48.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:48.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:48.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:48.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:48.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:49.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:48.877+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:49.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:48.877+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:49.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:48.877+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:49.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:49.083+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:49.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:49.083+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:49.303 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:49.083+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:49.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:49.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:49.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:49.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:49.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:49.478+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:50.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:49.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:50.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:49.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:50.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:49.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:50.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:50.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:50.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:50.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:50.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:50.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:50.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:50.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:50.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:50.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:50.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:50.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:51.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:50.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:51.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:50.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:51.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:50.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:51.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:51.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:51.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:51.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:51.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:51.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:51.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:51.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:51.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:51.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:51.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:51.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:52.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:51.870+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:52.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:51.870+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:52.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:51.870+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:52.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:52.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:52.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:52.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:52.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:52.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:52.791 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:15:52.662+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:15:52.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:52.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:52.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:52.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:52.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:52.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:53.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:52.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:53.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:52.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:53.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:52.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:53.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:53.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:53.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:53.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:53.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:53.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:53.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:53.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:53.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:53.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:53.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:53.417+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:54.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:53.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:54.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:53.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:54.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:53.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:54.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:54.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:54.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:54.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:54.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:54.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:54.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:54.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:54.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:54.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:54.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:54.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:55.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:55.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:55.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:54.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:54.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:54.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:55.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:55.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:55.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:55.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:55.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:55.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:56.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:56.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:56.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:55.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:55.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:55.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:56.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:56.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:56.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:56.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:56.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:56.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:57.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:56.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:57.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:56.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:57.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:56.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:57.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:57.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:57.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:57.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:57.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:57.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:57.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:57.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:57.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:57.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:57.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:57.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:58.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:15:57.663+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 49 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:15:58.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:57.889+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:58.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:57.889+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:58.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:57.889+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:58.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:58.025+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:58.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:58.025+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:58.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:58.025+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:58.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:58.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:58.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:58.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:58.802 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:58.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:58.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:58.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:58.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:59.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:59.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:59.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:15:59.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:15:59.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:59.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:15:59.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:15:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:00.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:59.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:00.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:59.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:00.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:15:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:15:59.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:00.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:00.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:00.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:00.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:00.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:00.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:00.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:00.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:00.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:00.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:00.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:00.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:01.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:00.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:01.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:00.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:01.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:00.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:01.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:01.062+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:01.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:01.062+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:01.304 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:01.062+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:01.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:01.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:01.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:01.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:01.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:01.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:02.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:01.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:02.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:01.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:02.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:01.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:02.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:02.029+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:02.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:02.029+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:02.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:02.029+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:02.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:02.664+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 74 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:02.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:02.525+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:02.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:02.525+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:02.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:02.525+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:03.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:02.921+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:03.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:02.921+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:03.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:02.921+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:03.517 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:03.064+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:03.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:03.064+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:03.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:03.064+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:03.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:03.547+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:03.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:03.547+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:03.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:03.547+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:04.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:03.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:04.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:03.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:04.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:03.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:04.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:04.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:04.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:04.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:04.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:04.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:04.996 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:04.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:04.996 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:04.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:04.996 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:04.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:05.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:04.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:05.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:04.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:05.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:04.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:05.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:05.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:05.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:05.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:05.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:05.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:05.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:05.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:05.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:05.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:05.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:05.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:05.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:05.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:05.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:06.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:06.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:06.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:06.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:07.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:06.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:07.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:06.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:07.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:06.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:07.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:07.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:07.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:07.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:07.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:07.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:07.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:07.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:07.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:07.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:07.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:07.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:07.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:07.665+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 104 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:07.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:07.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:07.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:07.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:07.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:07.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:08.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:08.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:08.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:08.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:08.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:08.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:08.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:08.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:08.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:08.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:08.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:08.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:08.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:08.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:09.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:08.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:09.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:09.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:09.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:09.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:09.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:09.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:09.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:10.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:09.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:10.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:09.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:10.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:09.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:10.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:10.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:10.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:10.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:10.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:10.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:10.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:11.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:10.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:11.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:10.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:11.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:10.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:11.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:11.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:11.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:11.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:11.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:11.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:11.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:12.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:11.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:12.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:11.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:12.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:11.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:12.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:12.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:12.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:12.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:12.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:12.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:12.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:12.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:12.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:12.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:12.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:12.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:13.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:12.666+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 131 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:13.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:12.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:13.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:12.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:13.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:12.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:13.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:13.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:13.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.017+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:13.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:13.074+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:13.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:13.074+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:13.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:13.074+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:13.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:13.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:13.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:13.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:14.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:14.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:14.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:14.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:14.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:14.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:14.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:14.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:14.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:14.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:14.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:14.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:14.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:15.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:15.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:15.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:15.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:15.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:15.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:15.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:15.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:15.981 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:15.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:15.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:15.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:15.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:16.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:16.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:16.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:16.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:17.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:16.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:17.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:16.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:17.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:16.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:17.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:17.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:17.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:17.112+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:17.112+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:17.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:17.112+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:17.792 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:17.666+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 161 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:17.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:17.635+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:17.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:17.635+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:17.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:17.635+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:18.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:18.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:18.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:18.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:18.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:18.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:18.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:18.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:18.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:18.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:19.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:19.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:19.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:19.129+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:19.129+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:19.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:19.129+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:20.019 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:19.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:20.019 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:19.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:20.019 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:19.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:20.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:20.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:20.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:20.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:20.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:20.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:20.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:20.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:20.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:20.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:20.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:20.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:21.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:20.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:21.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:20.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:21.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:20.614+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:21.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:21.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:21.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:21.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:21.111+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:21.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:21.111+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:21.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:21.111+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:21.961 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:21.961 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:21.961 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:22.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:22.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:22.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:21.961+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:22.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:22.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:22.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:22.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:22.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:22.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:22.983 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:22.667+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 190 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:22.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:22.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:22.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:22.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:22.984 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:22.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:23.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:22.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:23.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:22.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:23.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:22.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:23.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:23.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:23.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:23.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:23.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:23.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:23.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:23.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:23.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:23.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:23.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:23.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:24.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:23.999+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:24.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:23.999+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:24.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:23.999+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:24.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:24.094+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:24.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:24.094+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:24.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:24.094+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:24.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:24.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:24.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:24.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:24.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:24.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:24.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:24.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:24.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:25.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:25.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:25.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:25.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:25.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:25.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:25.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:25.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:25.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:25.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:25.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:25.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:26.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:25.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:26.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:26.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:26.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:26.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:26.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:26.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:26.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:26.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:26.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:26.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:26.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:26.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:27.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:27.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:27.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:27.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:27.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:27.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:27.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:27.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:27.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:27.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:27.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:27.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:27.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:27.668+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 219 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:27.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:27.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:27.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:27.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:27.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:27.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:28.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:28.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:28.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:28.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:28.051 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:28.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:28.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:28.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:28.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:28.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:28.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:28.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:29.036 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:28.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:29.036 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:28.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:29.036 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:28.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:29.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:29.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:29.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:29.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:29.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:29.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:29.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:30.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:29.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:30.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:29.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:30.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:29.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:30.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:30.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:30.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:30.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:30.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:30.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:30.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:30.992 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:30.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:30.992 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:30.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:30.992 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:30.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:30.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:31.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:31.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:31.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:31.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:31.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:31.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:31.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:31.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:31.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:31.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:31.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:31.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:32.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:31.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:32.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:31.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:32.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:31.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:32.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:32.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:32.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:32.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:32.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:32.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:32.793 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:32.668+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 246 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:32.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:32.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:32.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:32.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:32.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:32.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:33.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:32.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:33.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:32.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:33.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:32.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:33.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:33.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:33.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:33.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:33.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:33.078+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:33.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:33.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:33.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:33.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:33.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:33.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:34.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:34.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:34.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:34.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:34.115+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:34.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:34.115+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:34.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:34.115+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:34.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:34.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:34.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:34.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:34.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:34.522+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:35.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:35.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:35.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:35.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:35.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:35.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:35.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:35.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:35.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:34.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:35.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:35.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:35.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:35.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:35.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:35.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:36.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:35.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:36.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:35.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:36.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:35.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:36.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:36.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:36.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:36.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:36.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:36.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:36.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:36.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:36.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:36.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:36.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:36.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:37.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:36.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:37.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:36.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:37.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:36.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:37.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:37.070+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:37.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:37.070+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:37.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:37.070+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:37.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:37.485+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:37.669 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:37.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:37.670 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:37.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:37.992 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:37.669+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 276 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:38.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:37.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:38.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:37.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:38.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:37.993+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:38.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:38.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:38.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:38.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:38.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:38.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:38.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:38.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:38.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:39.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:39.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:39.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:38.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:39.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:39.546+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:39.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:39.546+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:39.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:39.546+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:40.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:40.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:40.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:39.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:40.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:39.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:40.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:39.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:40.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:39.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:40.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:40.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:40.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:40.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:40.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:40.512+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:41.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:40.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:41.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:40.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:41.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:40.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:41.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:40.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:41.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:40.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:41.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:40.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:41.302 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:40.971+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:41.993 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:41.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:41.994 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:41.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:41.994 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:41.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:42.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:41.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:42.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:41.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:42.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:41.994+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:42.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:41.994+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:42.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:42.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:42.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:42.996 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:42.670+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 302 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:42.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:42.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:42.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:42.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:42.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:42.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:43.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:43.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:43.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:43.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:43.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:43.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:43.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:43.021+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:43.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:43.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:43.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:42.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:43.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:43.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:43.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:44.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:44.033+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:44.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:44.033+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:44.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:44.033+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:44.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:44.033+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:44.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:43.950+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:44.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:43.950+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:44.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:43.950+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:44.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:44.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:44.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:44.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:44.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:44.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:45.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:45.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:45.067+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:45.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:44.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:45.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:44.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:45.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:44.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:45.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:45.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:45.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:45.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:45.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:45.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:46.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:46.067+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:46.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:46.067+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:46.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:46.067+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:46.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:46.067+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:46.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:45.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:46.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:45.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:46.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:45.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:47.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:46.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:47.003 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:46.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:47.003 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:46.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:47.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:47.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:47.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:47.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:47.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:47.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:47.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:47.106+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:47.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:47.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:47.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:47.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:47.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:47.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:47.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:47.670+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 332 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:47.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:47.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:47.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:47.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:47.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:47.559+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:48.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:48.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:48.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:48.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:48.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:48.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:48.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:48.142+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:48.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:48.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:48.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:48.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:48.521 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:48.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:48.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:48.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:48.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:48.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:48.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:48.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:49.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:49.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:49.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:49.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:49.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:49.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:49.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:49.162+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:49.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:49.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:49.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:49.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:49.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:49.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:50.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:49.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:50.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:49.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:50.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:49.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:50.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:50.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:50.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:50.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:50.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:50.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:50.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:50.124+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:50.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:50.085+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:50.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:50.085+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:50.551 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:50.085+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:51.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:51.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:51.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:51.138+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:51.138+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:51.138+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:51.138+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:51.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:51.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:51.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:51.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:51.044+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:52.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:51.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:52.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:51.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:52.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:51.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:52.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:52.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:52.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:52.168+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:52.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:52.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:52.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:52.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:52.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:52.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:53.013 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:52.671+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 359 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:53.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:52.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:53.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:52.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:53.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:52.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:53.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:53.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:53.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:53.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:53.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:53.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:53.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:53.150+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:53.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:53.150+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:53.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:53.150+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:53.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:53.150+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:53.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:53.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:53.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:53.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:53.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:53.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:54.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:54.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:54.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:54.134+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:54.134+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:54.134+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:54.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:54.134+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:55.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:54.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:55.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:54.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:55.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:54.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:55.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.023+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:55.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.023+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:55.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.023+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:55.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:55.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:55.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:55.124+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:55.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:55.124+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:55.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:55.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:55.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:55.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:55.997 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:55.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:56.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.998+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:56.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.998+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:56.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:55.998+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:56.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:56.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:56.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:56.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:56.107+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:56.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:56.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:56.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:56.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:56.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:56.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:57.137+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:57.137+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:57.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:57.137+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:57.301 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:57.137+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:57.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:16:57.672+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 392 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:16:57.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:57.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:57.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:57.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:57.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:57.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:58.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.012+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:58.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.012+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:58.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.012+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:58.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:58.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:58.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:58.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:58.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:58.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:58.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:58.148+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:58.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:58.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:58.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:58.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:58.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:58.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.976+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.976+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:58.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:59.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:59.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:59.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:16:59.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:16:59.171+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:16:59.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:59.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:16:59.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:16:59.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:00.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:00.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:00.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:00.147+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:00.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:59.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:00.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:59.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:00.301 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:16:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:16:59.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:00.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:00.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:00.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:00.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:00.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:00.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:01.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:00.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:01.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:00.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:01.187 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:00.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:01.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:01.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:01.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:01.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:01.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:01.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:01.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:01.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:01.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:02.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:01.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:02.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:01.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:02.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:01.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:02.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:02.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:02.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:02.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:02.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:02.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:02.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:02.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:02.196+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:03.050 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:02.673+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 420 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:03.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:02.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:03.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:02.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:03.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:02.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:03.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:03.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:03.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:03.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:03.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:03.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:03.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:03.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:03.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:03.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:03.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:03.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:03.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:03.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:04.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:03.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:04.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:03.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:04.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:03.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:04.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:04.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:04.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:04.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:04.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:04.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:04.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:04.283+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:05.232 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:04.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:05.232 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:04.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:05.233 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:04.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:05.390+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:05.390+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:05.390+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:05.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:05.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:05.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:05.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:05.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:05.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:06.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:05.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:06.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:05.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:06.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:05.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:06.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:06.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:06.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:06.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:06.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:06.550 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:06.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:06.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:06.190+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:07.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:07.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:07.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:07.511 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:07.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:07.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:07.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:07.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:07.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:07.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:07.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:07.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:07.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:07.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:07.160+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:07.799 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:07.673+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 449 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:08.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:07.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:08.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:07.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:08.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:07.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:08.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:08.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:08.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:08.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:08.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:08.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:08.398 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:08.145+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:08.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:08.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:08.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:08.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:08.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:08.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:09.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:08.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:09.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:08.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:09.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:08.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:09.393 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:09.126+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:09.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:09.126+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:09.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:09.126+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:09.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:09.126+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:09.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:09.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:09.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:09.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:09.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:09.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:10.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:09.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:10.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:09.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:10.140 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:09.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:10.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:10.140+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:10.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:10.141+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:10.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:10.141+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:10.394 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:10.141+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:10.766 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:10.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:10.766 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:10.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:10.766 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:10.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:11.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:10.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:11.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:10.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:11.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:10.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:11.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:11.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:11.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:11.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:11.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:11.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:11.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:11.099+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:11.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:11.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:11.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:11.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:11.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:11.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:12.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:11.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:12.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:11.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:12.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:11.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:12.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:12.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:12.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:12.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:12.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:12.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:12.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:12.103+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:12.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:12.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:12.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:12.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:13.050 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:12.674+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 476 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:13.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:12.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:13.052 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:12.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:13.052 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:12.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:13.362+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:13.362+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:13.362+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:13.131+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:13.131+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:13.131+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:13.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:13.131+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:13.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:13.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:13.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:13.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:13.800 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:13.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:14.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:14.088+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:14.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:14.088+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:14.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:14.088+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:14.347 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:14.088+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:14.750 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:14.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:14.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:14.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:14.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:14.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:15.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:14.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:15.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:14.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:15.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:14.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:15.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:15.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:15.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:15.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:15.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:15.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:15.318 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:15.063+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:15.730 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:15.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:15.730 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:15.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:15.730 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:15.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:16.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:16.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:16.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:16.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:16.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:16.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:16.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:16.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:16.063+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:16.358 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:16.063+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:16.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:16.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:16.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:16.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:16.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:16.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:17.027 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:16.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:17.028 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:16.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:17.028 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:16.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:17.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:17.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:17.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:17.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:17.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:17.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:17.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:17.029+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:17.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:17.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:17.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:17.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:17.550 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:17.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:18.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:17.675+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 509 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:18.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:17.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:18.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:17.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:18.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:17.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:18.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:18.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:18.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:18.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:18.333 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:18.051+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:18.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:18.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:18.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:18.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:18.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:18.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:19.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:18.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:19.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:18.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:19.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:18.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:19.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:19.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:19.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:19.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:19.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:19.060+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:19.380 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:19.060+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:19.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:19.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:19.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:19.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:19.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:19.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:20.102 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:19.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:20.102 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:19.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:20.102 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:19.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:20.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:20.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:20.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:20.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:20.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:20.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:20.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:20.103+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:20.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:20.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:20.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:20.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:20.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:20.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:21.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:20.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:21.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:20.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:21.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:20.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:21.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:21.082+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:21.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:21.082+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:21.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:21.082+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:21.354 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:21.082+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:21.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:21.356+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:21.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:21.356+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:21.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:21.356+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:22.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:21.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:22.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:21.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:22.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:21.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:22.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:22.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:22.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:22.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:22.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:22.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:22.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:22.035+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:22.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:22.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:22.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:22.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:22.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:22.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:23.050 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:22.676+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 537 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:23.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:22.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:23.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:22.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:23.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:22.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:23.367 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:23.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:23.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:23.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:23.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:23.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:23.368 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:23.084+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:23.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:23.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:23.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:23.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:23.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:23.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:24.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:23.865+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:24.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:23.865+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:24.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:23.865+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:24.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:24.098+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:24.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:24.098+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:24.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:24.098+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:24.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:24.098+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:24.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:24.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:24.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:24.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:24.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:24.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:25.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:24.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:24.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:24.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:25.064+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:25.064+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:25.064+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:25.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:25.064+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:25.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:25.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:25.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:25.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:25.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:25.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:26.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:25.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:25.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:25.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:26.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:26.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:26.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:26.042+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:26.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:26.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:26.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:26.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:26.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:26.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:26.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:26.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:26.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:27.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:27.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:27.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:27.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:27.032+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:27.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:27.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:27.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:27.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:27.675 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:27.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:28.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:27.676+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 566 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:28.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:27.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:28.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:27.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:28.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:27.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:28.076+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:28.077+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:28.077+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:28.473 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:28.077+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:28.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:28.474+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:28.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:28.474+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:28.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:28.474+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:29.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:28.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:29.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:28.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:29.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:28.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:29.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:29.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:29.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:29.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:29.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:29.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:29.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:29.028+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:29.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:29.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:29.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:29.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:29.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:29.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:30.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:29.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:29.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:29.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:30.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:30.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:30.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:30.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:30.027+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:30.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:30.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:30.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:30.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:30.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:30.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:30.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:30.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:30.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:31.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:31.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:31.043+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:31.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:31.043+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:31.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:31.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:31.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:31.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:32.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:31.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:32.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:31.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:32.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:31.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:32.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:32.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:32.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:32.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:32.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:32.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:32.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:32.044+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:32.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:32.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:32.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:32.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:32.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:32.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:33.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:32.677+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 593 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:33.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:32.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:33.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:32.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:33.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:32.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:33.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:33.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:33.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:33.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:33.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:33.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:33.432 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:33.055+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:33.756 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:33.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:33.756 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:33.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:33.756 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:33.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:34.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:33.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:34.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:33.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:34.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:33.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:34.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:34.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:34.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:34.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:34.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:34.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:34.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:34.054+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:34.715 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:34.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:34.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:34.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:34.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:34.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:35.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:34.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:35.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:34.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:35.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:34.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:35.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:35.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:35.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:35.020+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:35.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:35.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:35.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:35.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:36.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:35.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:36.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:35.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:36.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:35.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:36.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:36.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:36.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:36.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:36.022+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:36.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:36.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:36.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:36.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:36.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:36.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:37.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:37.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:37.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:37.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:37.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:37.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:37.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:37.004+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:37.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:36.776+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:37.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:36.776+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:37.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:36.776+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:37.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:37.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:37.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:37.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:37.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:37.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:38.026 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:37.678+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 623 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:38.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:37.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:38.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:37.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:38.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:37.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:38.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:38.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:38.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:38.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:38.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:38.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:38.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:38.027+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:38.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:38.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:38.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:38.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:38.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:38.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:38.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:38.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:38.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:39.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.039+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:39.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:39.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:39.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:39.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:39.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:39.410+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:40.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:39.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:40.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:39.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:40.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:39.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:40.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:40.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:40.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:40.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:39.991+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:40.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:40.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:40.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:40.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:40.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:40.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:41.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:40.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:41.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:40.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:41.302 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:40.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:41.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:41.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:41.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:41.303 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:41.016+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:41.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:41.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:41.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:41.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:41.634 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:41.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:42.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:41.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:42.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:41.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:42.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:41.752+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:42.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:42.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:42.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:42.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:42.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:42.020+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:42.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:42.020+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:42.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:42.354+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:42.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:42.354+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:42.678 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:42.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:43.030 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:42.678+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 649 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:43.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:42.790+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:43.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:42.790+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:43.031 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:42.790+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:43.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:43.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:43.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:43.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:43.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:43.032+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:43.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:43.032+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:43.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:43.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:43.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:43.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:44.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:44.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:44.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:44.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:44.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:44.035+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:44.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:44.036+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:44.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:43.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:44.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:43.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:44.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:43.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:44.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:44.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:44.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:44.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:44.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:44.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:45.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:44.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:45.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:44.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:45.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:44.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:45.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:45.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:45.066+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:45.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:45.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:45.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:45.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:45.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:45.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:46.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:46.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:46.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:46.055+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:46.055+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:45.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:45.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:46.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:45.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:46.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:46.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:46.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:47.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:47.048+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:47.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:47.048+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:47.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:47.048+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:47.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:47.048+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:47.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:46.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:47.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:46.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:47.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:46.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:47.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:47.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:47.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:47.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:47.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:47.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:48.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:47.679+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 677 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:48.049 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:47.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:48.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:47.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:48.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:47.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:48.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:48.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:48.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:48.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:48.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:48.066+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:48.436 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:48.066+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:48.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:48.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:48.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:48.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:48.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:48.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:49.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:48.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:49.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:48.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:49.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:48.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:49.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:49.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:49.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.030+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:49.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.030+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:49.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:49.397+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:49.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:49.397+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:49.800 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:49.397+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:49.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:49.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:49.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:50.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:49.995+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:50.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:50.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:50.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:50.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:50.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:50.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:51.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:50.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:51.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:50.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:51.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:50.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:51.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:51.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:51.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:51.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:51.021+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:51.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:51.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:51.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:51.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:51.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:51.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:52.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:51.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:52.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:51.919+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:52.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:51.919+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:52.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:52.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:52.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:52.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:52.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:52.007+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:52.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:52.680+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 705 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:52.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:52.431+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:52.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:52.431+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:52.693 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:52.431+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:53.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:52.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:53.027 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:52.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:53.027 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:52.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:53.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:53.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:53.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:53.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:53.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:53.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:53.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:53.028+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:53.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:53.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:53.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:53.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:53.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:53.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:54.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:53.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:54.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:53.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:54.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:53.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.007+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:54.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.007+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:54.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:54.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:54.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:54.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:55.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:54.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:55.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:54.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:55.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:54.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:55.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:55.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:55.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:55.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:54.975+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:55.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:55.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:55.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:55.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:55.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:55.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:56.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:55.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:56.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:55.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:56.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:55.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:56.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:55.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:56.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:55.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:56.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:55.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:56.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:55.943+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:56.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:56.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:56.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:56.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:56.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:56.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:57.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:56.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:57.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:56.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:57.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:56.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:57.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:56.938+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:56.938+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:56.938+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:56.938+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:57.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:57.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:57.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:57.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:57.679 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:57.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:57.952 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:17:57.680+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 734 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:17:57.952 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:57.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:57.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:57.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:57.953 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:57.917+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:58.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:57.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:58.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:57.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:58.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:57.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:58.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:58.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:58.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:58.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:58.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:58.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:59.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:58.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:59.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:58.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:59.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:58.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:17:59.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:58.955+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:17:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:58.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:58.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:59.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:58.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:17:59.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:59.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:59.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:59.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:17:59.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:17:59.328+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:00.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:59.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:00.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:59.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:00.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:17:59.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:00.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:59.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:59.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:59.918+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:00.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:17:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:17:59.918+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:00.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:00.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:00.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:00.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:00.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:00.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:00.963+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:00.963+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:00.963+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:00.897+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:00.897+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:00.897+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:01.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:00.897+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:01.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:01.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:01.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:01.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:01.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:01.291+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:02.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:01.892+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:01.892+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:01.892+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:01.892+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:02.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:01.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:02.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:01.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:02.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:01.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:02.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:02.340+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:02.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:02.340+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:02.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:02.340+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:02.958 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:02.681+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 765 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:02.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:02.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:02.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:02.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:02.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:02.894+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:03.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:02.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:03.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:02.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:03.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:02.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:03.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:03.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:03.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:03.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:03.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:03.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:03.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:03.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:03.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:03.917+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:04.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:03.958+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:04.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:03.958+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:04.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:03.958+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:04.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:04.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:04.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:04.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:04.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:04.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:04.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:04.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:04.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:04.929+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:04.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:04.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:05.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:04.997+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:05.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:05.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:05.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:05.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:05.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:05.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:06.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:05.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:06.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:05.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:06.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:05.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:06.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:05.887+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:06.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:05.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:06.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:05.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:06.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:05.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:06.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:06.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:06.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:06.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:06.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:06.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:06.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:06.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:06.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:06.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:06.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:06.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:07.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:06.936+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:07.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:07.357+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:07.680 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:07.357+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:07.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:07.357+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:07.963 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:07.682+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 802 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:07.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:07.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:07.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:07.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:07.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:07.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:07.966+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:08.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:08.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:08.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:08.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:08.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:08.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:08.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:08.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:08.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:08.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:08.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:09.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:08.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:09.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:08.966+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:09.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:09.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:09.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:09.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:09.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:09.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:09.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:09.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:10.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:09.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:10.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:09.984+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:10.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:10.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:10.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:10.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:10.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:10.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:10.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:11.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:11.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:11.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:11.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:11.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:11.027+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:11.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:11.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:11.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:11.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:11.635 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:11.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:12.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:12.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:12.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:12.026+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:11.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:12.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:11.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:12.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:11.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:12.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:12.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:12.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:12.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:12.681 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:12.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:12.946 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:12.683+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 830 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:12.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:12.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:12.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:13.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:13.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:13.037+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:13.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:13.037+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:13.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:13.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:13.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:13.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:13.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:13.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:14.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:14.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:14.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:14.024+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:13.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:13.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:13.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:14.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:14.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:14.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:14.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:14.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:14.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:14.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:14.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:14.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:15.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:15.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.002+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:15.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:15.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:15.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:15.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:15.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:15.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:15.979+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:15.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:16.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:15.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:16.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:15.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:16.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:16.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:16.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:16.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:16.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:16.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:16.934+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:17.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:17.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:17.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:17.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:17.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:17.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:17.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:17.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:17.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:18.034 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:17.683+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 859 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:18.034 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:17.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:18.034 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:17.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:18.034 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:17.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:18.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:17.924+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:18.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:18.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:18.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:18.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:18.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:18.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:18.732 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:18.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:18.732 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:18.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:18.732 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:18.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:19.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:18.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:19.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:18.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:19.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:18.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:19.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:18.902+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:19.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:19.050+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:19.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:19.050+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:19.422 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:19.050+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:19.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:19.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:19.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:19.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:19.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:19.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:19.876+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:19.876+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:19.876+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:19.876+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:20.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:20.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:20.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:20.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:20.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:20.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:20.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:20.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:20.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:20.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:20.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:21.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:20.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:21.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:20.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:21.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:20.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:21.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:20.893+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:21.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:21.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:21.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:21.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:21.300 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:21.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:21.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:21.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:21.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:21.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:21.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:21.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:22.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:21.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:22.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:21.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:22.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:21.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:22.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:21.844+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:22.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:22.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:22.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:22.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:22.413 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:22.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:22.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:22.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:22.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:22.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:22.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:22.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:23.049 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:22.684+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 886 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:23.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:22.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:23.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:22.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:23.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:22.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:23.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:22.837+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:23.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:23.122+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:23.442 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:23.122+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:23.443 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:23.122+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:23.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:23.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:23.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:23.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:23.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:23.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:24.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:23.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:24.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:23.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:24.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:23.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:24.133 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:23.883+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:24.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:24.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:24.437 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:24.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:24.438 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:24.135+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:24.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:24.439+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:24.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:24.439+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:24.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:24.439+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:25.121+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:25.121+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:25.121+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:24.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:24.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:24.929+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:25.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:24.929+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:25.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:25.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:25.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:25.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:25.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:25.468+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:26.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:26.099+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:26.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:26.099+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:26.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:26.099+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:26.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:25.931+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:26.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:25.931+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:25.931+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:26.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:25.931+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:26.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:26.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:26.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:26.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:26.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:26.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:26.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:26.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:26.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:26.958+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:27.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:27.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:27.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:27.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:27.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:27.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:27.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:27.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:27.683 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:27.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:27.948 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:27.685+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 914 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:28.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:28.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:28.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:28.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:27.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:27.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:27.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:28.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:27.950+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:28.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:28.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:28.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:28.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:28.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:28.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:29.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:29.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:29.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:29.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:28.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:28.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:28.987+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:29.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:28.987+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:29.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:29.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:29.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:29.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:29.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:29.366+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:30.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:30.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:30.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:30.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:29.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:29.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:29.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:30.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:29.988+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:30.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:30.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:30.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:30.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:30.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:30.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:31.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:31.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:31.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:31.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:31.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:31.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:31.026+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:31.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:31.027+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:31.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:31.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:31.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:31.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:31.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:31.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:32.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:32.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:32.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:32.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:32.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:32.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:32.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:32.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:32.040+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:32.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:32.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:32.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:32.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:32.684 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:32.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:33.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:32.685+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 942 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:33.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:33.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:33.407 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:33.407 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:33.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:33.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:33.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:33.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:33.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:33.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:33.058+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:33.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:33.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:33.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:33.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:33.409+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:34.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:34.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:34.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:34.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:34.024+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:34.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:34.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:34.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:34.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:34.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:34.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:34.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:34.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:34.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:34.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:35.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.031+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:35.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:35.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:35.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:35.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:35.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:35.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:35.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:35.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:35.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:35.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:36.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.982+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.982+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.982+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:35.982+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:36.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:36.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:36.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:36.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:36.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:36.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:36.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:36.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:36.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:36.416+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:37.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:36.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:36.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:36.995+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:36.995+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:37.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:37.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:37.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:37.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:37.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:37.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:37.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:37.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:37.685 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:37.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:37.955 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:37.686+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 971 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:38.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:37.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:38.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:37.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:38.210 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:37.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:38.211 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:37.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:38.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:38.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:38.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:38.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:38.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:38.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:38.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:38.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:38.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:38.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:38.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:38.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:39.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:39.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:39.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:38.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:38.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:38.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:39.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:38.969+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:39.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:39.352+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:39.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:39.352+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:39.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:39.352+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:40.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:40.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:40.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:40.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:39.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:39.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:39.949+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:40.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:39.949+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:40.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:40.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:40.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:40.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:40.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:40.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:41.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:41.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:41.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:40.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:40.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:40.983+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:41.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:40.983+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:41.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:41.288+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:41.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:41.288+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:41.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:41.288+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:42.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:41.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:42.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:41.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:42.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:41.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:42.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:41.973+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:42.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:42.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:42.265+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:42.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:42.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:42.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:42.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:43.012 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:42.687+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1000 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:43.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:43.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:43.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:43.285 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:43.286 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.014+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:43.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:43.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:43.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:43.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:43.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:43.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:44.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:44.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.974+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:44.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.974+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:44.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:43.974+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:44.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:44.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:44.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:44.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:44.321+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:44.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:44.321+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:44.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:44.321+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:45.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:44.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:45.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:44.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:45.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:44.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:45.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:44.934+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:45.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:45.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:45.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:45.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:45.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:45.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:45.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:45.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:45.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:45.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:45.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:45.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:46.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:46.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:46.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:46.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:46.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:46.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:46.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:45.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:46.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:45.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:46.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:45.972+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:46.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:45.973+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:46.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:46.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:46.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:46.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:46.333+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:47.249 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:46.953+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:47.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:46.953+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:47.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:46.953+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:47.250 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:46.953+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:47.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:47.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:47.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:47.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:47.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:47.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:47.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:47.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:47.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:47.688+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1033 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:48.213 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:47.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:48.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:47.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:48.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:47.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:48.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:47.966+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:48.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:48.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:48.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:48.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:48.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:48.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:48.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:48.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:48.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:48.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:48.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:48.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:49.244 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:48.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:49.244 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:48.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:49.244 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:48.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:49.244 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:48.986+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:49.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:49.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:49.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:49.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:49.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:49.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:49.245 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:18:49.245+0000 7f8ab7bd0700 0 monclient(hunting): authenticate timed out after 300 2023-12-19T19:18:49.245 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-19T19:18:49.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:49.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:49.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:49.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:49.548 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:49.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:49.928 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2023-12-19T19:18:50.207 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:18:50.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:49.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:50.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:49.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:50.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:49.980+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:50.236 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:49.980+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:50.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:50.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:50.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:50.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:50.517 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:50.371+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:50.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:50.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:50.518 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:50.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:51.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:50.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:51.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:50.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:51.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:50.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:51.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:50.990+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:51.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:51.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:51.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:51.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:51.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:51.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:52.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:51.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:52.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:51.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:52.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:51.958+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:52.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:51.958+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:52.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:52.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:52.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:52.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:52.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:52.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:52.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:52.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:52.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:52.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:52.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:52.688+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1066 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:53.234 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:52.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:53.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:52.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:53.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:52.936+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:53.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:52.936+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:53.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:53.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:53.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:53.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:53.526 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:53.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:53.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:53.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:53.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:53.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:53.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:53.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:54.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:53.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:54.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:53.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:54.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:53.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:54.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:53.889+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:54.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:54.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:54.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:54.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:54.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:54.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:54.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:54.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:54.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:54.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:54.549 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:54.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:55.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:54.841+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:55.185 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:54.841+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:55.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:54.841+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:55.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:54.841+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:55.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:55.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:55.467 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:55.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:55.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:55.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:55.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:55.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:55.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:55.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:55.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:55.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:56.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:55.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:56.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:55.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:56.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:55.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:56.141 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:55.882+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:56.490 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:56.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:56.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:56.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:56.491 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:56.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:56.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:56.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:56.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:56.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:56.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:56.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:57.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:56.908+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:57.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:56.908+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:57.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:56.908+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:57.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:56.908+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:57.496 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:57.173+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:57.496 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:57.173+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:57.496 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:57.173+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:57.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:18:57.689+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1100 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:18:57.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:57.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:57.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:57.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:57.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:57.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:58.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:57.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:58.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:57.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:58.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:57.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:58.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:57.906+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:58.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:58.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:58.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:58.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:58.527 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:58.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:58.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:58.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:58.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:58.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:58.799 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:58.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:59.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:58.858+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:59.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:58.858+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:59.148 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:58.858+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:18:59.149 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:58.858+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:18:59.522 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:59.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:59.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:59.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:59.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:18:59.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:18:59.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:59.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:59.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:59.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:18:59.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:18:59.526+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:00.108+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:00.108+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:00.108+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:59.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:59.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:00.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:59.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:00.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:18:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:18:59.874+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:00.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:00.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:00.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:01.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:01.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:01.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:01.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:01.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:01.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:01.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:00.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:01.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:00.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:01.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:00.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:01.300 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:00.919+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:01.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:01.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:01.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:01.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:01.959 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:01.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:02.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:02.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:02.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:02.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:02.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:02.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:02.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:01.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:02.246 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:01.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:02.246 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:01.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:02.246 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:01.962+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:02.941 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:02.690+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1132 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:02.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:02.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:02.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:02.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:02.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:02.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:03.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:03.072+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:03.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:03.072+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:03.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:03.072+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:03.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:02.944+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:03.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:02.944+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:03.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:02.944+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:03.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:02.944+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:03.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:03.616+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:03.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:03.616+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:03.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:03.616+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:04.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:04.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:04.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:04.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:04.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:04.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:03.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:03.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:03.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:04.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:03.954+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:04.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:04.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:04.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:04.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:04.947 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:04.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:04.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:04.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:04.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:04.950+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:05.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:05.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:05.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:05.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:05.107+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:05.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:05.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:05.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:05.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:05.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:05.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:06.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:05.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:06.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:05.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:06.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:05.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:06.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:05.967+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:06.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:06.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:06.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:06.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:06.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:06.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:06.990 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:06.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:06.990 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:06.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:06.990 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:06.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:07.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:06.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:07.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:06.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:07.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:06.992+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:07.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:06.992+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:07.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:07.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:07.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:07.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:07.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:08.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:07.691+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1165 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:08.038 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:07.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:08.038 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:07.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:08.038 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:07.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:08.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:08.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:08.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:08.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:08.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:08.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:08.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:08.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:08.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:08.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:08.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:08.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:08.040+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:09.010 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:08.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:09.010 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:08.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:09.010 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:08.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:09.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:09.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:09.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:09.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:09.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:09.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:09.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:09.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:09.012+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:09.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:09.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:09.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:09.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:09.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:09.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:10.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:10.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:10.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:10.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:10.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:10.165+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:10.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:10.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:10.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:10.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:10.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:10.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:10.299 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:10.045+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:11.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:10.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:11.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:10.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:11.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:10.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:11.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:11.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:11.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:11.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:11.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:11.090+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:11.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:11.090+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:11.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:11.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:11.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:11.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:11.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:11.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:12.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:11.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:12.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:11.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:12.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:11.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:12.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:12.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:12.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:12.081+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:12.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:12.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:12.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:12.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:13.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:12.691+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1198 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:13.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:12.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:13.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:12.635+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:13.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:12.635+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:13.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:13.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:13.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:13.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:13.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:13.050+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:13.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:13.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:13.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:13.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:13.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:13.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:13.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:13.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:13.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:14.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:14.046+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:14.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:14.046+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:14.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:14.046+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:14.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:14.046+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:14.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:14.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:14.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:14.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:14.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:14.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:15.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:14.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:15.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:15.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:15.029+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:15.029+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:15.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:15.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:15.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:15.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:15.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:15.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:15.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:15.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:15.801 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:15.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:16.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:16.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:16.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:16.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:16.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:16.021+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:16.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:16.022+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:16.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:16.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:16.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:16.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:16.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:16.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:17.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:16.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:17.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:16.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:17.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:16.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:17.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:17.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:17.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:17.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:17.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:17.027+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:17.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:17.028+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:17.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:17.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:17.299 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:17.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:17.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:17.692+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1232 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:17.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:17.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:17.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:17.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:17.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:17.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:18.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:18.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:18.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:18.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:18.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:18.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:18.215 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:18.050+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:18.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:18.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:18.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:18.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:18.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:18.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:19.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:18.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:19.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:18.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:19.006 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:18.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:19.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:19.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:19.242+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:19.073+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:19.073+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:19.073+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:19.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:19.074+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:19.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:19.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:20.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:19.602+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:20.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:20.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:20.223+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:20.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:20.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:20.106+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:20.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:20.106+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:21.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:20.631+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:21.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:20.631+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:21.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:20.631+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:21.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:21.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:21.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:21.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:21.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:21.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:21.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:21.155+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:22.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:21.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:22.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:21.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:22.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:21.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:22.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:22.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:22.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:22.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:22.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:22.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:22.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:22.183+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:23.048 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:22.693+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1264 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:23.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:22.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:23.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:22.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:23.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:22.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:23.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:23.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:23.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:23.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:23.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:23.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:23.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:23.192+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:23.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:23.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:23.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:23.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:23.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:23.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:23.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:23.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:23.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:24.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:24.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:24.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:24.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:24.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:24.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:24.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:24.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:24.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:24.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:24.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:24.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:24.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:24.165+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:25.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:24.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:25.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:24.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:25.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:24.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:25.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:25.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:25.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:25.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:25.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:25.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:25.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:25.182+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:26.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:25.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:26.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:25.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:26.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:25.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:26.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:26.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:26.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:26.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:26.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:26.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:26.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:26.160+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:27.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:26.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:27.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:26.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:27.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:26.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:27.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:27.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:27.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:27.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:27.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:27.289+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:27.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:27.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:27.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:27.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:27.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:27.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:27.511 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:27.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:27.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:27.694+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1297 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:27.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:27.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:27.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:27.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:27.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:27.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:28.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:28.269+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:28.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:28.269+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:28.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:28.269+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:28.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:28.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:28.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:28.204+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:28.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:28.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:28.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:28.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:28.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:28.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:29.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:29.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:29.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:29.222+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:29.222+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:29.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:29.222+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:29.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:29.222+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:29.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:29.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:29.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:29.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:29.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:29.497+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:30.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:30.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:30.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:30.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:30.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:30.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:30.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:30.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:30.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:30.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:30.532 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:30.211+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:30.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:30.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:30.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:30.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:30.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:30.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:31.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:31.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:31.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:31.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:31.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:31.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:31.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:31.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:31.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:31.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:31.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:31.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:31.549 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:31.194+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:32.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:31.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:32.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:31.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:32.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:31.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:32.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:32.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:32.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:32.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:32.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:32.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:32.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:32.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:32.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:32.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:32.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:32.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:32.146+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:32.717 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:32.695+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1389 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:32.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:32.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:32.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:32.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:32.718 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:32.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:33.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:33.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:33.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:33.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:33.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:33.099+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:33.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:33.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:33.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:33.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:34.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:34.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:34.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:34.140+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:34.140+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:34.140+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:34.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:34.140+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:34.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:35.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:34.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:35.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:34.596+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:35.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:35.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:35.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:35.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:35.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:35.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:35.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:35.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:35.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:35.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:35.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:35.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:35.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:35.172+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:35.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:35.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:35.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:35.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:35.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:35.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:36.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:36.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:36.142+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:36.142+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:36.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:36.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:36.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:36.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:37.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:36.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:37.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:36.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:37.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:36.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:37.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:37.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:37.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:37.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:37.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:37.165+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:37.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:37.165+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:37.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:37.696+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1473 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:37.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:37.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:37.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:38.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:38.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:38.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:38.201+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:38.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:38.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:38.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:38.549 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:38.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:39.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:38.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:39.067 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:38.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:39.068 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:38.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:39.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:39.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:39.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:39.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:39.232+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:39.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:39.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:39.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:39.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:40.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:39.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:40.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:39.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:40.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:39.587+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:40.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:40.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:40.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:40.248+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:40.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:40.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:40.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:40.183+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:40.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:40.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:40.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:41.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:41.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:41.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:41.296+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:41.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:41.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:41.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:41.156+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:42.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:41.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:42.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:41.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:42.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:41.573+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:42.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:42.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:42.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:42.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:42.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:42.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:42.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:42.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:42.267+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:42.818 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:42.697+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1555 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:42.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:42.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:42.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:42.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:42.818 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:42.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:43.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:43.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:43.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:43.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:43.528 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:43.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:43.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:43.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:43.529 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:43.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:43.529 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:43.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:43.529 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:43.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:43.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:43.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:43.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:43.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:43.591+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:44.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:44.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:44.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:44.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:44.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:44.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:44.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:44.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:44.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:45.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:44.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:45.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:44.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:45.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:44.555+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:45.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:45.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:45.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:45.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:45.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:45.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:45.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:45.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:45.267+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:46.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:45.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:46.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:45.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:46.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:45.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:46.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:46.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:46.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:46.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:46.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:46.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:46.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:46.247+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:47.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:46.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:47.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:46.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:47.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:46.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:47.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:47.200+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:47.200+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:47.200+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:47.200+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:47.066+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:47.066+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:47.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:47.066+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:47.798 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:47.698+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1638 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:47.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:47.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:47.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:47.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:47.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:47.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:48.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:48.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:48.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:48.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:48.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:48.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:48.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:48.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:48.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:48.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:48.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:48.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:48.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:48.202+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:49.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:48.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:49.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:48.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:49.013 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:48.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:49.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:49.016+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:49.016+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:49.016+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:49.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:49.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:49.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:49.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:49.185+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:49.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:49.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:49.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:49.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:49.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:49.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:50.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:50.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:50.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:50.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:50.173+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:50.173+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:50.173+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:50.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:50.173+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:51.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:50.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:51.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:50.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:51.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:50.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:51.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:51.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:51.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:51.183+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:51.183+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:51.183+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:51.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:51.183+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:52.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:51.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:52.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:51.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:52.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:51.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:52.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:52.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:52.056+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:52.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:52.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:52.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:52.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:52.217+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:53.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:52.698+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1721 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:53.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:52.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:53.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:52.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:53.025 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:52.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:53.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:53.297 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:53.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:53.298 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:53.226+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:53.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:53.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:53.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:53.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:53.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:53.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:53.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:53.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:53.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:53.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:53.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:53.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:54.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:54.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:54.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:54.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:54.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:54.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:54.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:54.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:54.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:54.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:54.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:54.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:54.262+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:55.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:54.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:55.032 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:54.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:55.032 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:54.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:55.286 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:55.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:55.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:55.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:55.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:55.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:55.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:55.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:55.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:55.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:55.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:55.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:55.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:55.290+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:55.581+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:55.581+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:55.581+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:56.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:56.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:56.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:56.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:56.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:56.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:56.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:56.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:56.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:56.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:56.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:56.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:56.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:56.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:57.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:56.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:57.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:56.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:57.002 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:56.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:57.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:57.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:57.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.006+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:57.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:57.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:57.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:57.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:57.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:57.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:57.548 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:57.280+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:57.823 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:19:57.700+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1805 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:19:57.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:57.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:57.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:57.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:57.823 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:57.569+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:58.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:58.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:58.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:57.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:58.528+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:58.528+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:58.528+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:58.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:58.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:58.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:58.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:58.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:19:59.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:59.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:59.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:59.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:59.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:19:59.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:19:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:59.483+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:59.483+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:19:59.483+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:19:59.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:59.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:59.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:19:59.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:19:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:19:59.350+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:00.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:00.042+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:00.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:00.042+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:00.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:00.042+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:00.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:00.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:00.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:00.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:00.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:00.372+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:00.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:00.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:00.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:00.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:01.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:01.034+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:01.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:01.034+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:01.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:01.034+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:01.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:01.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:01.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:01.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:01.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:01.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:01.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:01.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:01.640 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:01.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:01.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:01.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:01.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:01.383+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:02.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:02.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:02.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:02.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:02.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:02.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:02.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:02.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:02.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:02.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:02.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:02.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:02.997 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:02.700+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1887 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:03.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:03.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:03.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:03.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:03.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:03.000+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:03.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:03.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:03.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:03.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:03.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:03.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:03.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:03.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:03.396+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:04.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:04.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:04.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:04.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:04.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:04.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:04.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:04.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:04.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:04.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:04.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:04.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:04.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:04.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:04.370+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:05.393 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:05.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:05.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:05.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:05.394 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:05.051+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:05.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:05.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:05.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:05.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:05.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:05.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:05.395 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:05.397+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:05.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:05.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:05.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:05.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:05.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:05.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:06.348 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:06.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:06.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:06.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:06.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:06.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:06.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:06.349 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:06.351+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:06.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:06.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:06.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:07.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:07.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:07.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:07.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:07.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:07.048+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:07.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:07.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:07.553 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:07.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:07.554 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:07.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:07.554 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:07.377+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:07.824 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:07.701+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 1970 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:07.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:07.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:07.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:07.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:07.824 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:07.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:08.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:08.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:08.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:08.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:08.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:08.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:08.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:08.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:08.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:08.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:08.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:08.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:08.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:08.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:08.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:08.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:08.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:08.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:08.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:08.361+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:09.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:09.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:09.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:09.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:09.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:09.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:09.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:09.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:09.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:09.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:09.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:09.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:09.342+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:10.312 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:10.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:10.312 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:10.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:10.313 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:10.063+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:10.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:10.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:10.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:10.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:10.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:10.523+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:10.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:10.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:10.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:10.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:10.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:10.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:10.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:10.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:11.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:11.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:11.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:11.640 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:11.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:11.641 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:11.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:11.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:11.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:11.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:11.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:11.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:11.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:11.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:11.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:12.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.985+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:12.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.985+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:12.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:11.985+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:12.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:12.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:12.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:12.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:12.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:12.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:12.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:12.340+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:12.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:12.702+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2053 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:13.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:12.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:13.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:12.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:13.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:12.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:13.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:13.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:13.582+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:13.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:13.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:13.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:13.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:13.799 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:13.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:13.987+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:13.987+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:13.987+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:14.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:14.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:14.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:14.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:14.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:14.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:14.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:14.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:14.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:14.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:14.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:14.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:14.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:14.376+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:15.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:14.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:15.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:14.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:15.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:14.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:15.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:15.570+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:15.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:15.570+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:15.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:15.570+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:15.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:15.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:15.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:15.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:15.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:15.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:15.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:15.394+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:15.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:15.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:15.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:16.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:16.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:16.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:16.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:16.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:16.544+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:16.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:16.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:16.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:16.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:16.364+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:17.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:17.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:17.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:17.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:17.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:17.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:17.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:17.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:17.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:17.700 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:17.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:17.700 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:17.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:17.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:17.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:17.701 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:17.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:17.976 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:17.703+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2137 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:18.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:17.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:18.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:17.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:18.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:17.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:18.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:18.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:18.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:18.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:18.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:18.519+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:18.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:18.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:18.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:18.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:18.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:18.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:18.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:18.432+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:19.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:18.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:19.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:18.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:19.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:18.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:19.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:19.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:19.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:19.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:19.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:19.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:19.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:19.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:19.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:19.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:19.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:19.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:19.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:19.386+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:19.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:19.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:19.969+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:20.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:20.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:20.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:20.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:20.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:20.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:20.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:20.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:20.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:20.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:20.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:20.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:20.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:20.340+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:21.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:20.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:21.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:20.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:21.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:20.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:21.595 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:21.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:21.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:21.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:21.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:21.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:21.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:21.305+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:21.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:21.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:21.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:21.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:22.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:21.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:22.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:21.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:22.277 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:21.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:22.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:22.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:22.280+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:22.280+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:22.826 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:22.704+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2219 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:22.826 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:22.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:22.826 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:22.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:22.826 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:22.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:23.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:22.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:23.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:22.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:23.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:22.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:23.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:23.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:23.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:23.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:23.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:23.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:23.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:23.326+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:23.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:23.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:23.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:23.540+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:24.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:23.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:24.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:23.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:24.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:23.975+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:24.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:24.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:24.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:24.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:24.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:24.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:24.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:24.320+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:25.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:25.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:25.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:25.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:25.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:25.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:25.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:25.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:25.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:25.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:25.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:25.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:25.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:25.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:25.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:25.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:25.318+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:25.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:25.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:26.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:25.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:26.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:26.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:26.599 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:26.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:26.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:26.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:26.600 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:26.343+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:26.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:26.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:26.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:26.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:27.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:27.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:27.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:27.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:27.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:27.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:27.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:27.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:27.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:27.703 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:27.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:27.987 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:27.705+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2302 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:28.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:28.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:28.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:27.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:28.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:28.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:28.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:28.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:28.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:28.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:28.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:28.356+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:29.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:28.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:29.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:28.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:29.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:28.657+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:29.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:29.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:29.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:29.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:29.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:29.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:29.687 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:29.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:29.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:29.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:29.688 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:29.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:30.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:29.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:30.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:29.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:30.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:29.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:30.416 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:30.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:30.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:30.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:30.417 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:30.082+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:30.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:30.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:30.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:30.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:30.671 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:30.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:30.672 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:30.420+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:31.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:30.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:30.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:30.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:31.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:31.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:31.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:31.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:31.117+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:31.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:31.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:31.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:31.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:31.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:31.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:31.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:31.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:31.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:31.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:31.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:31.429+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:32.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:32.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:32.472 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:32.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:32.728 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:32.706+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2385 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:32.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:32.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:32.617+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:32.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:32.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:32.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:32.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:32.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:33.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:33.045+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:33.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:33.045+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:33.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:33.045+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:33.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:33.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:33.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:33.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:33.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:33.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:33.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:33.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:33.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:33.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:33.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:34.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:34.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:34.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:34.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:34.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:34.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:34.646+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:34.646+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:34.646+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:34.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:34.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:35.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:34.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:35.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:34.556+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:35.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:35.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:35.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:35.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:35.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:35.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:35.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:35.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:35.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:35.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:35.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:35.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:35.592+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:36.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:36.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:36.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:36.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:36.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:36.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:36.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:36.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:36.585+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:36.970 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:36.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:36.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:36.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:36.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:36.589+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:37.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:37.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:37.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:37.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:37.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:37.125+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:37.797 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:37.707+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2469 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:37.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:37.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:37.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:37.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:37.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:37.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:37.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:37.605+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:37.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:37.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:37.798 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:37.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:38.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:38.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:38.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:38.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:38.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:38.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:38.593+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:38.593+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:38.593+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:38.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:38.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:38.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:39.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:38.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:39.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:39.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:39.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:39.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:39.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:39.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:40.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:39.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:40.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:39.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:40.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:39.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:40.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:39.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:40.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:39.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:40.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:39.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:40.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:39.668+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:40.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:40.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:40.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:40.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:40.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:40.128+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:41.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:40.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:41.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:40.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:41.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:40.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:41.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:40.714+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:40.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:40.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:41.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:40.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:41.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:41.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:41.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:41.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:41.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:41.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:41.649+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:41.649+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:41.649+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:41.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:41.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:42.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:41.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:42.048 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:41.730+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:42.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:42.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:42.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:42.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:42.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:42.073+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:43.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:42.708+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2551 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:42.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:42.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:42.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:42.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:42.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:42.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:42.682+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:43.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:43.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:43.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:43.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:43.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:43.046+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:43.664+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:43.664+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:43.664+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:43.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:43.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:43.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:43.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:43.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:43.689+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:44.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:44.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:44.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:44.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:44.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:44.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:45.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:44.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:44.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:44.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:44.713+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:44.713+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:44.713+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:45.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:44.713+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:45.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:45.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:45.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:45.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:45.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:45.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:46.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:45.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:45.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:45.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:45.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:45.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:45.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:46.023 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:45.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:46.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:46.026+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:46.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:46.026+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:46.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:46.026+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:47.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:46.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:47.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:46.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:47.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:46.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:47.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:47.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:47.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:47.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:46.727+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:47.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:47.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:47.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:47.035+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:47.765 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:47.710+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2634 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:47.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:47.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:47.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:47.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:47.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:47.642+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:48.049 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:47.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:48.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:47.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:48.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:47.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:48.050 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:47.770+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:48.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:48.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:48.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:48.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:48.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:48.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:49.035 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:48.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:49.035 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:48.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:49.035 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:48.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:49.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:48.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:49.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:48.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:49.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:48.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:49.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:48.775+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:49.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:49.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:49.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:49.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:49.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:49.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:50.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:49.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:50.007 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:49.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:50.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:49.600+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:50.008 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:49.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:50.008 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:49.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:50.008 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:49.734+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:50.008 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:49.734+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:50.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:50.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:50.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:50.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:50.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:50.011+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:50.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:50.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:50.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:50.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:51.047 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:50.728+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:51.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:51.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:51.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:51.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:51.547 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:51.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:52.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:51.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:52.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:51.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:52.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:51.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:52.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:51.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:52.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:51.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:52.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:51.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:52.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:51.698+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:52.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:52.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:52.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:52.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:52.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:52.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:52.710+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2717 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:52.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:52.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:52.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:52.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.016 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:52.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:52.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:52.651+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:53.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:53.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:53.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:53.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:53.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:53.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:53.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:53.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:53.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:53.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:53.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:53.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:53.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:53.610+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:54.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:54.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:54.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.028+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:54.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:54.723+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:54.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:54.723+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:54.976 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:54.723+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:54.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:54.598+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:54.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:54.598+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:54.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:54.598+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:54.977 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:54.598+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:55.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:55.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:55.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:54.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:55.691 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:55.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:55.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:55.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:55.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:55.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:55.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:55.568+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:55.989 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:55.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:55.990 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:55.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:55.990 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:55.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:56.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:55.993+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:56.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:55.993+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:56.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:55.993+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:56.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:56.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:56.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:56.535+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:56.535+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:56.535+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:56.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:56.535+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:57.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:56.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:57.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:56.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:57.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:56.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:57.797 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:20:57.711+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2800 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:20:57.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:57.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:57.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:57.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:57.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:57.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:57.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:57.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:57.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:57.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:57.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:57.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:57.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:57.534+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:58.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:58.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:58.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:58.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:58.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:58.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:58.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:58.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:58.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:58.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:58.523+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:20:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:58.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:59.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:58.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:59.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:58.952+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:59.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:59.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:20:59.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:59.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:59.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:59.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:20:59.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:20:59.517+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:00.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:59.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:00.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:59.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:00.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:20:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:20:59.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:00.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:00.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:00.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:00.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:00.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:00.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:00.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:00.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:01.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:00.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:01.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:00.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:01.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:00.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:01.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:01.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:01.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:01.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:01.480+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:01.480+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:01.480+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:01.480+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:01.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:01.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:01.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:02.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:02.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:02.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:02.458+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:02.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:02.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:02.709 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:02.675+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:03.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:02.712+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2881 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:03.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:02.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:03.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:02.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:03.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:02.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:03.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:03.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:03.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:03.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:03.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:03.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:03.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:03.462+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:04.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:03.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:04.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:03.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:04.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:03.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:04.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:03.714+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:04.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:03.714+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:04.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:03.714+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:04.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:04.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:04.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:04.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:04.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:04.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:04.740 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:04.483+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:05.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:04.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:05.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:04.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:05.050 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:04.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:05.050 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:04.743+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:05.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:04.743+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:05.051 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:04.743+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:05.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:05.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:05.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:05.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:05.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:05.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:05.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:05.447+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:05.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:05.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:05.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:06.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:06.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:06.452+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:06.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:06.452+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:06.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:06.453+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:06.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:06.453+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:07.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:06.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:07.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:06.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:07.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:06.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:07.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:06.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:07.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:06.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:07.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:06.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:07.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:07.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:07.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:07.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:07.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:07.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:07.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:07.427+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:08.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:07.713+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 2964 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:08.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:07.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:08.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:07.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:08.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:07.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:08.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:07.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:08.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:07.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:08.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:07.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:08.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:08.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:08.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:08.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:08.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:08.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:08.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:08.420+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:09.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:08.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:09.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:08.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:09.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:08.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:09.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:08.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:09.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:08.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:09.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:08.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:09.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:09.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:09.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:09.404+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:10.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:09.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:10.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:09.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:10.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:09.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:10.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:10.298 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:10.298 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:10.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:10.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:10.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:10.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:10.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:10.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:10.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:10.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:10.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:10.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:10.909+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:10.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:10.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:11.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:10.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:11.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:11.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:11.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:11.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:11.410+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:12.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:11.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:12.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:11.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:12.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:11.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:12.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:11.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:12.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:11.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:12.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:11.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:12.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:12.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:12.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:12.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:12.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:13.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:12.714+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3046 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:12.936+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:12.936+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:12.936+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:12.902+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:12.902+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:13.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:12.902+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:13.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:13.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:13.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:13.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:13.392+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:13.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:13.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:13.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:13.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:13.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:14.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:13.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:14.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:14.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:14.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:14.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:14.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:14.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:14.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:14.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:15.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:14.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:15.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:14.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:15.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:14.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:15.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:14.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:15.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:14.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:15.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:14.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:15.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:15.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:15.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:15.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:15.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:15.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:15.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:15.454+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.009+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.009+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.009+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:16.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:16.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:16.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:16.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:16.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:16.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:17.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.961+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:17.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.961+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:17.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:16.961+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:17.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:16.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:17.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:16.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:17.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:16.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:17.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:17.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:17.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:17.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:17.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:17.395+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:17.967 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:17.715+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3131 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:17.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:17.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:17.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:17.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:17.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:17.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:18.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:17.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:18.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:17.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:18.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:17.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:18.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:18.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:18.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:18.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:18.359+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:19.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:18.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:19.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:18.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:19.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:18.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:19.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:18.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:19.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:18.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:19.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:18.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:19.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:19.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:19.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:19.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:19.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:19.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:19.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:19.327+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:20.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:19.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:19.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:19.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:20.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:20.014+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:20.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:20.014+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:20.297 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:20.014+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:20.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:20.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:20.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:20.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:20.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:20.316+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:20.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:20.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:21.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:21.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:21.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:21.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:21.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:21.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:21.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:20.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:21.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:20.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:21.271 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:20.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:21.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:21.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:21.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:21.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:21.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:21.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:21.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:21.274+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:22.008 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:21.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:22.008 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:21.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:22.009 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:21.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:22.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:22.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:22.259 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:22.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:22.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:22.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:22.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:22.262+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:22.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:22.263+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:22.963 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:22.716+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3212 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:22.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:22.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:22.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:22.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:22.964 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:22.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:23.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:23.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:23.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:22.968+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:23.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:23.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:23.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:23.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:23.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:23.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:23.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:23.273+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:24.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:23.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:24.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:23.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:24.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:23.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:24.235 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:23.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:24.235 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:23.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:24.235 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:23.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:24.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:24.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:24.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:24.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:24.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:24.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:24.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:24.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:25.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:24.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:25.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:24.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:25.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:24.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:25.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:24.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:25.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:24.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:25.258 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:24.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:25.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:25.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:25.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:25.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:25.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:25.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:25.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:25.261+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:26.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:25.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:25.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:26.223 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:25.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:25.889+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:25.889+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:26.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:25.889+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:26.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:26.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:26.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:26.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:26.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:26.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:26.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:26.227+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:27.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:26.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:27.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:26.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:27.211 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:26.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:27.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:26.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:27.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:26.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:27.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:26.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:27.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:27.216+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:27.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:27.216+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:27.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:27.216+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:27.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:27.216+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:27.796 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:27.717+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3296 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:27.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:27.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:27.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:27.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:27.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:28.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:27.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:28.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:28.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:28.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:28.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:28.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:28.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:28.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:28.245+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:29.193 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:28.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:29.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:28.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:29.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:28.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:29.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:28.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:29.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:28.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:29.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:28.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:29.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:29.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:29.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:29.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:29.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:29.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:29.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:29.236+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:30.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:29.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:30.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:29.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:30.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:29.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:30.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:29.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:30.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:29.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:30.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:29.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:30.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:30.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:30.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:30.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:30.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:30.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:30.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:30.232+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:31.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:30.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:31.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:30.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:31.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:30.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:31.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:30.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:31.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:30.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:31.248 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:30.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:31.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:31.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:31.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:31.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:31.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:31.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:31.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:31.252+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:31.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:31.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:31.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:32.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:31.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:32.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:31.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:32.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:31.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:32.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:32.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:32.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:32.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:32.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:32.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:32.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:32.245+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:33.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:32.718+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3378 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:33.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:32.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:33.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:32.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:33.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:32.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:33.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:32.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:33.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:32.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:33.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:32.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:33.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:33.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:33.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:33.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:33.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:33.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:33.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:33.235+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:34.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:33.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:34.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:33.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:34.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:33.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:34.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:33.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:34.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:33.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:34.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:33.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:34.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:34.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:34.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:34.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:34.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:34.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:34.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:34.209+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:35.194 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:34.807+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:35.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:34.807+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:35.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:34.807+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:35.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:34.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:35.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:34.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:35.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:34.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:35.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:35.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:35.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:35.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:35.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:35.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:35.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:35.199+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:35.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:35.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:35.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:35.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:35.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:36.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:35.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:36.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:36.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:36.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:36.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:36.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:36.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:36.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:36.176+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:37.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:36.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:37.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:36.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:37.154 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:36.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:37.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:36.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:37.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:36.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:37.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:36.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:37.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:37.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:37.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:37.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:37.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:37.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:37.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:37.158+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:37.796 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:37.719+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3463 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:38.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:37.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:38.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:37.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:38.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:37.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:38.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:37.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:38.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:37.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:38.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:37.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:38.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:38.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:38.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:38.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:38.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:38.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:38.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:38.168+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:38.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:38.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:38.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:38.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:38.814+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:39.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:38.814+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:39.219+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:39.219+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:39.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:39.219+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:39.547 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:39.219+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:40.182 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:39.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:40.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:39.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:40.183 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:39.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:40.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:39.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:40.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:39.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:40.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:39.904+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:40.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:40.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:40.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:40.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:40.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:40.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:40.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:40.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:41.232 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:40.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:41.232 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:40.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:41.232 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:40.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:41.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:40.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:41.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:40.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:41.233 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:40.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:41.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:41.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:41.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:41.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:41.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:41.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:41.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:41.236+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:42.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:41.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:42.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:41.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:42.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:41.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:41.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:41.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:42.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:41.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:42.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:42.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:42.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:42.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:42.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:42.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:42.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:42.226+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:43.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:42.720+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3544 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:43.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:42.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:42.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:42.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:42.775+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:43.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:42.775+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:43.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:42.775+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:43.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:43.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:43.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:43.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:43.216+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:43.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:43.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:43.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:43.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:43.740+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:43.740+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:43.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:43.740+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:44.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:44.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:44.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:44.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:44.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:44.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:44.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:44.187+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:44.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:44.744+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:44.804 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:44.744+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:44.805 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:44.744+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:45.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:44.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:45.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:44.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:45.207 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:44.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:45.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:45.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:45.211+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:45.211+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:46.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:45.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:46.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:45.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:46.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:45.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:46.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:46.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:46.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:46.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:46.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:46.227+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:46.228+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:47.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:46.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:47.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:46.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:47.048 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:46.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:47.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:46.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:47.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:46.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:47.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:46.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:47.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:47.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:47.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:47.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:47.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:47.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:47.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:47.194+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:47.782 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:47.721+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3628 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:48.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:47.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:48.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:47.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:48.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:47.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:48.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:47.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:48.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:47.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:48.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:47.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:48.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:48.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:48.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:48.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:48.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:48.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:48.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:48.236+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:48.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:48.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:48.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:49.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:48.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:49.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:48.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:49.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:48.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:49.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:49.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:49.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:49.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:49.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:49.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:49.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:49.187+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:50.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:49.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:50.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:49.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:50.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:49.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:50.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:49.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:50.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:49.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:50.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:49.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:50.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:50.233+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:50.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:50.233+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:50.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:50.233+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:50.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:50.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:51.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:50.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:51.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:50.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:51.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:50.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:51.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:50.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:51.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:50.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:51.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:50.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:51.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:51.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:51.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:51.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:51.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:51.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:51.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:51.254+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:51.692+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:51.692+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:51.898 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:51.692+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:52.272 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:51.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:52.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:51.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:52.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:51.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:52.276+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:52.276+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:52.276+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:52.277+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:53.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:52.722+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3710 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:52.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:52.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:52.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:52.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:52.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:53.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:52.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:53.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:53.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:53.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:53.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:53.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:53.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:53.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:53.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:53.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:53.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:53.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:53.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:53.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:53.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:54.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:53.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:54.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:53.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:54.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:53.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:54.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:54.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:54.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:54.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:54.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:54.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:54.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:54.329+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:54.941 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:54.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:54.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:54.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:54.942 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:54.671+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:55.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:54.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:55.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:54.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:55.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:54.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:55.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:55.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:55.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:55.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:55.362+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:55.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:55.653+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:55.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:55.653+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:55.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:55.653+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:56.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:55.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:56.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:55.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:56.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:55.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:56.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:56.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:56.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:56.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:56.608 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:56.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:56.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:56.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:56.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:56.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:56.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:56.362+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:56.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:56.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:57.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:56.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:57.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:56.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:57.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:56.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:57.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:57.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:57.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:57.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:57.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:57.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:57.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:57.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:57.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:57.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:58.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:21:57.723+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3795 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:21:58.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:57.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:58.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:57.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:58.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:57.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:58.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:58.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:58.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:58.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:58.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:58.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:58.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:58.422+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:58.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:58.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:58.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:58.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:58.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:58.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:59.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:58.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:21:59.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:59.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:59.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:59.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:59.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:59.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:21:59.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:21:59.386+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:21:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:59.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:59.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:21:59.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:21:59.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:00.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:59.818+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:00.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:59.818+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:00.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:21:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:21:59.818+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:00.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:00.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:00.572+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:00.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:00.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:00.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:00.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:00.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:00.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:01.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:00.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:01.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:00.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:01.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:00.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:01.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:01.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:01.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:01.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:01.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:01.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:01.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:01.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:01.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:01.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:01.419+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:02.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:01.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:02.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:01.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:02.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:01.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:02.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:02.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:02.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:02.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:02.721 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:02.455+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:02.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:02.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:02.721 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:02.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:02.984 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:02.724+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3876 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:02.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:02.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:02.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:02.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:02.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:02.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:03.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:03.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:03.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:03.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:03.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:03.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:03.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:03.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:03.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:03.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:03.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:03.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:03.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:03.422+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:04.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:03.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:04.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:03.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:04.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:03.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:04.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:04.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:04.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:04.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:04.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:04.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:04.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:04.452+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:04.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:04.452+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:04.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:04.452+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:04.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:04.452+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:05.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:04.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:05.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:04.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:05.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:04.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:05.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:05.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:05.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:05.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:05.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:05.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:05.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:05.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:05.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:05.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:05.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:05.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:05.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:05.486+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:06.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:05.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:06.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:05.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:06.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:05.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:06.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:06.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:06.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:06.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:06.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:06.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:06.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:06.512+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:06.567+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:06.567+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:06.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:06.567+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:07.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:06.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:07.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:06.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:07.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:06.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:07.831 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:07.725+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 3960 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:07.831 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:07.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:07.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:07.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:07.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:07.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:07.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:07.832 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:07.554+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:08.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:07.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:08.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:07.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:08.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:07.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:08.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:08.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:08.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:08.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:08.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:08.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:08.879 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:08.564+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:09.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:08.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:09.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:08.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:09.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:08.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:09.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:09.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:09.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:09.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:09.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:09.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:09.604+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:09.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:09.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:09.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:09.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:09.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:09.527+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:10.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:09.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:10.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:09.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:10.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:09.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:10.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:10.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:10.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:10.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:10.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:10.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:10.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:10.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:10.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:10.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:10.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:10.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:10.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:10.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:11.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:10.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:11.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:10.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:11.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:10.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:11.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:11.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:11.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:11.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:11.892 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:11.566+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:11.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:11.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:11.893 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:11.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:11.893 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:11.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:11.893 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:11.626+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:11.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:11.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:12.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:11.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:12.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:12.726+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4046 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:12.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:12.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:12.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:12.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:12.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:12.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:12.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:12.595+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:13.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:12.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:13.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:12.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:13.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:12.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:13.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:13.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:13.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:13.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:13.797 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:13.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:13.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:13.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:13.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:13.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:13.798 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:13.631+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:14.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:13.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:14.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:13.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:14.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:13.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:14.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:14.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:14.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:14.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:14.872 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:14.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:14.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:14.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:14.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:14.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:14.592+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:15.272 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:14.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:15.272 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:14.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:15.272 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:14.877+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:15.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:15.709+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:15.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:15.709+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:15.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:15.709+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:15.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:15.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:15.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:15.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:15.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:15.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:15.888 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:15.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:16.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:15.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:16.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:15.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:16.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:15.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:16.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:16.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:16.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:16.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:16.905 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:16.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:16.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:16.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:16.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:16.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:16.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:16.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:16.906 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:16.619+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:17.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:16.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:17.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:16.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:17.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:16.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:17.796 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:17.728+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4135 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:17.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:17.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:17.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:17.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:17.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:17.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:17.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:17.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:17.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:17.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:17.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:17.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:17.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:17.590+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:18.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:17.916+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:18.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:17.916+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:18.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:17.916+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:18.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:18.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:18.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:18.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:18.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:18.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:18.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:18.599+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:18.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:18.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:18.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:18.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:18.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:18.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:19.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:19.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:19.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.908 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:19.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:19.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:19.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:19.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:19.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:19.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:19.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:19.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:19.583+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:20.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:19.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:20.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:19.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:20.297 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:19.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:20.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:20.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:20.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:20.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:20.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:20.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:20.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:20.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:20.601+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:21.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:20.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:21.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:20.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:21.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:20.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:21.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:21.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:21.610+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:21.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:21.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:21.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:21.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:21.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:22.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:21.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:22.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:21.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:22.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:21.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:22.850 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:22.729+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4222 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:22.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:22.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:22.594+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:22.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:22.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:22.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:22.851 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:22.618+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:23.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:22.951+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:23.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:22.951+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:23.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:22.951+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:23.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:23.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:23.613+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:23.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:23.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:23.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:23.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:23.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:24.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:23.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:24.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:23.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:24.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:23.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:24.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:24.597+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:24.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:24.597+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:24.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:24.597+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:24.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:24.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:24.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:24.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:24.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:24.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:24.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:24.577+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:25.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:24.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:25.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:24.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:25.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:24.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:25.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:25.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:25.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:25.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:25.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:25.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:25.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:25.539+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:26.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:26.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:26.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:26.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:26.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:26.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:26.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:26.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:26.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:26.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:26.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:27.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:27.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:27.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:27.747 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:27.730+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4311 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:27.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:27.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:27.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:27.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:27.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:27.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:27.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:27.547+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:27.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:27.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:27.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:28.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:28.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:28.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:28.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:28.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:28.533+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:28.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:28.533+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:29.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:28.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:29.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:28.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:29.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:28.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:29.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:29.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:29.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:29.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:29.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:29.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:29.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:29.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:29.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:29.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:29.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:29.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:29.797 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:29.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:30.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:29.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:30.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:29.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:30.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:29.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:30.514+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:30.514+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:30.514+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:30.514+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:30.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:30.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:30.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:30.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:31.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:30.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:31.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:30.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:31.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:30.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:31.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:31.365+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:31.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:31.365+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:31.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:31.365+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:31.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:31.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:31.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:31.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:31.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:32.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:32.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:32.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:32.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:32.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:32.036+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:32.726 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:32.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:32.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:32.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:32.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:32.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:32.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:32.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:32.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:33.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:32.731+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4393 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:33.402 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:33.076+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:33.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:33.076+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:33.403 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:33.076+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:33.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:33.407+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:33.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:33.407+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:33.704 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:33.407+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:33.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:33.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:33.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:33.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:33.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:33.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:33.705 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:33.428+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:34.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:34.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:34.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:34.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:34.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:34.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:34.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:34.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:34.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:34.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:35.362 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:35.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:35.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:35.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:35.363 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:35.075+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:35.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:35.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:35.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:35.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:35.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:35.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:35.505+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:36.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:36.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:36.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:36.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:36.049+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:36.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:36.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:36.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:36.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:36.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:37.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:37.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:37.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:37.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:37.331 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:37.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:37.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:37.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:37.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:37.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:37.445+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:38.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:37.732+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4492 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:38.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:38.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:38.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:38.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:38.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:38.022+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:38.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:38.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:38.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:38.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:38.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:38.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:38.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:38.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:39.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:39.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:39.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:39.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:39.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:39.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:39.346+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:39.346+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:39.346+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:39.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:39.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:39.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:39.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:40.017+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:40.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:40.017+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:40.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:40.017+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:40.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:40.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:40.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:40.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:40.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:40.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:40.365+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:41.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:41.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:41.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:41.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:41.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:41.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:41.650 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:41.392+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:41.392+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:41.392+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:41.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:41.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:41.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:41.651 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:41.343+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:42.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.003+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:42.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.004+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:42.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.004+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:42.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:42.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:42.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:42.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:42.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:42.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:42.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:42.344+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:43.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:42.733+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4573 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:43.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:43.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:43.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:42.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:43.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:43.435+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:43.435+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:43.435+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:43.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:43.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:43.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:43.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:43.358+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:43.974+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:44.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:43.974+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:44.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:43.974+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:44.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:44.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:44.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:44.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:44.308+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:44.308+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:44.308+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:44.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:44.309+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:45.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:44.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:45.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:44.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:45.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:44.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:45.502+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:45.502+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:45.502+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:45.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:45.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:45.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:45.298+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:45.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:45.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:45.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:46.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:46.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:46.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:46.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:46.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:46.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:46.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:46.552+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:46.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:46.552+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:46.930 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:46.552+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:47.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:46.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:47.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:46.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:47.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:46.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:47.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:47.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:47.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:47.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:47.597 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:47.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:47.597 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:47.336+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:47.857 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:47.734+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4657 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:47.857 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:47.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:47.857 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:47.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:47.857 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:47.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:48.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:47.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:48.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:47.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:48.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:47.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:48.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:48.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:48.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:48.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:48.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:48.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:48.635 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:48.358+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:48.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:48.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:48.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:48.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:48.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:49.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:48.640+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:49.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:49.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:49.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:49.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:49.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:49.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:49.616 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:49.349+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:49.887 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:49.621+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:49.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:49.621+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:49.888 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:49.621+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:50.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:49.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:50.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:49.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:50.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:49.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:50.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:50.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:50.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:50.615+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:50.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:50.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:50.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:50.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:50.386+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:51.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:50.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:51.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:50.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:51.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:50.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:51.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:51.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:51.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:51.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:51.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:51.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:51.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:51.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:51.433+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:52.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:51.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:52.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:51.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:52.136 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:51.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:52.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:52.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:52.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:52.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:52.389+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:52.859 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:52.736+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4744 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:52.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:52.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:52.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:52.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:52.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:52.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:52.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:53.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:53.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:53.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:53.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:53.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:53.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:53.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:53.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:53.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:53.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:53.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:53.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:53.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:53.342+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:54.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:53.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:54.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:53.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:54.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:53.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:54.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:54.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:54.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:54.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:54.372+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:54.714 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:54.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:54.714 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:54.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:54.714 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:54.620+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:55.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:54.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:55.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:54.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:55.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:54.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:55.366+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:55.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:55.366+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:55.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:55.366+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:55.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:55.366+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:55.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:56.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:55.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:56.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:55.660+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:56.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:55.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:56.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:55.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:56.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:55.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:56.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:56.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:56.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:56.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:56.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:56.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:56.647 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:56.357+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:57.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:56.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:56.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:56.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:57.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:56.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:56.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:56.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:57.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:57.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:57.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:57.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:57.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:57.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:57.405+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:57.796 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:22:57.737+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4847 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:22:57.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:57.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:57.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:57.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:57.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:57.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:57.797 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:57.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:57.797 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:57.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:58.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:58.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:58.634+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:58.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:58.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:58.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:58.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:58.401+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:22:59.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:58.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:59.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:58.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:59.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:58.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:22:59.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:59.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:59.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:59.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:59.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:59.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:22:59.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:22:59.396+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:00.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:59.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:00.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:59.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:00.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:22:59.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:00.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:59.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:00.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:59.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:00.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:22:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:22:59.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:00.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:00.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:00.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:00.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:00.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:01.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:01.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:01.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:01.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:00.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:01.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:00.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:01.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:00.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:01.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:01.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:01.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:01.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:01.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:01.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:01.619 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:01.370+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:02.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:01.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:02.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:01.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:02.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:01.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:02.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:01.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:01.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:01.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.733 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:02.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:02.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:02.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:02.393+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:02.393+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:02.393+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:02.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:02.393+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:03.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:02.738+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 4941 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:03.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:02.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:03.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:02.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:03.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:02.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:03.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:03.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:03.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:03.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:03.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:03.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:03.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:03.399+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:04.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:03.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:04.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:03.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:04.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:03.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:04.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:03.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:04.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:03.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:04.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:03.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:04.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:04.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:04.648 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:04.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:04.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:04.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:04.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:04.377+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:05.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:04.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:05.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:04.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:05.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:04.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:05.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:04.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:05.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:04.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:05.047 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:04.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:05.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:05.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:05.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:05.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:05.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:05.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:05.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:05.382+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:06.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:05.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:06.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:05.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:06.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:05.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:06.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:05.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:06.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:05.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:06.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:05.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:06.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:06.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:06.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:06.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:06.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:06.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:06.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:06.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:07.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:06.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:07.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:06.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:07.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:06.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:07.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:06.789+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:07.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:06.789+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:07.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:06.789+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:07.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:07.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:07.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:07.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:07.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:07.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:07.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:07.382+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:07.739+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5025 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:07.749+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:07.749+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:07.749+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:07.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:08.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:07.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:08.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:07.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:08.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:08.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:08.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:08.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:08.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:08.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:08.368+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:09.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:08.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:09.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:08.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:09.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:08.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:09.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:08.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:09.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:08.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:09.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:08.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:09.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:09.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:09.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:09.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:09.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:09.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:09.690 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:09.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:10.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:09.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:10.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:09.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:10.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:09.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:10.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:10.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:10.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:09.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:10.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:10.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:10.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:10.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:10.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:10.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:10.684 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:10.401+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:11.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:10.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:11.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:10.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:11.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:10.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:11.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:10.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:11.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:10.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:11.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:10.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:11.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:11.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:11.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:11.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:11.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:11.410+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:12.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:11.736+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:12.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:11.736+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:12.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:11.736+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:12.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:11.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:12.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:11.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:12.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:11.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:12.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:12.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:12.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:12.710 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:12.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:13.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:12.741+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5107 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:12.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:12.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:12.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:12.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:12.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:13.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:12.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:13.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:13.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:13.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:13.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:13.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:13.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:13.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:13.412+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:14.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:13.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:14.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:13.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:14.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:13.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:14.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:13.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:14.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:13.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:14.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:13.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:14.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:14.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:14.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:14.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:14.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:14.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:14.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:14.400+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:15.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:14.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:15.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:14.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:15.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:14.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:15.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:14.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:15.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:14.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:15.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:14.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:15.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:15.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:15.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:15.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:15.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:15.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:15.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:15.365+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:16.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:15.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:16.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:15.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:16.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:15.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:16.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:15.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:16.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:15.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:16.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:15.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:16.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:16.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:16.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:16.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:16.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:16.364+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:17.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:16.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:17.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:16.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:17.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:16.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:17.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:17.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:17.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:17.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:17.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:17.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:17.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:17.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:17.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:17.702 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:17.359+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:18.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:17.742+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5192 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:17.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:17.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:17.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:17.708+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:17.708+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:18.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:17.708+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:18.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:18.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:18.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:18.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:18.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:18.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:18.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:18.354+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:18.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:18.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:18.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:18.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:18.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:19.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:18.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:19.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:19.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:19.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:19.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:19.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:19.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:19.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:19.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:20.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:19.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:20.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:19.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:20.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:19.801+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:20.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:19.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:20.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:19.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:20.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:19.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:20.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:20.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:20.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:20.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:20.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:20.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:20.681 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:20.403+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:21.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:20.817+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:21.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:20.817+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:21.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:20.817+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:21.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:20.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:21.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:20.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:21.047 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:20.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:21.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:21.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:21.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:21.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:21.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:21.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:21.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:21.408+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:22.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:21.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:22.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:21.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:22.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:21.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:22.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:21.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:22.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:21.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:22.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:21.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:22.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:22.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:22.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:22.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:22.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:22.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:22.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:22.367+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:23.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:22.743+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5273 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:22.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:22.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:22.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:22.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:22.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:23.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:22.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:23.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:23.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:23.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:23.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:23.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:23.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:23.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:23.411+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:24.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:23.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:24.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:23.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:24.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:23.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:24.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:23.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:24.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:23.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:24.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:23.713+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:24.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:24.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:24.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:24.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:24.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:24.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:24.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:24.399+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:25.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:24.782+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:24.782+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:24.782+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:24.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:25.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:24.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:25.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:24.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:25.644 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:25.385+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:25.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:25.385+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:25.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:25.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:25.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:25.386+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:25.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:25.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:25.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:25.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:25.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:25.928 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:25.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:26.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:26.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:26.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:26.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:26.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:26.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:26.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:26.351+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:27.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:26.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:27.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:26.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:27.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:26.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:27.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:26.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:27.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:26.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:27.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:26.742+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:27.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:27.325+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:27.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:27.325+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:27.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:27.325+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:27.711 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:27.325+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:27.744+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5357 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:28.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:27.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:28.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:27.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:28.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:27.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:27.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:27.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:28.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:27.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:28.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:28.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:28.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:28.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:28.346+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:29.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:28.748+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:29.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:28.748+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:29.001 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:28.748+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:29.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:28.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:29.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:28.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:29.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:28.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:29.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:29.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:29.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:29.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:29.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:29.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:29.685 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:29.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:30.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:29.702+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:30.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:29.702+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:30.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:29.702+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:29.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:29.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:30.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:29.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:30.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:30.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:30.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:30.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:30.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:30.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:30.649 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:30.398+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:31.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:30.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:31.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:30.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:31.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:30.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:31.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:30.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:31.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:30.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:31.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:30.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:31.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:31.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:31.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:31.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:31.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:31.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:31.606 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:31.352+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:31.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:31.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:31.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:31.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:31.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:32.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:31.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:32.741 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.741 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.741 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:32.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:32.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:32.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:32.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:32.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:32.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:32.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:32.369+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:33.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:32.745+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5439 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:32.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:32.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:32.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:33.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:33.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:33.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:33.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:33.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:33.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:33.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:33.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:33.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:33.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:33.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:33.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:33.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:33.391+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:34.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:33.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:34.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:33.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:34.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:33.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:34.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:34.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:34.501+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:34.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:34.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:34.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:34.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:34.422+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:35.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:34.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:35.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:34.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:35.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:34.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:35.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:35.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:35.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:35.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:35.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:35.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:35.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:35.414+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:35.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:35.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:35.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:36.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:36.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:36.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:36.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:36.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:36.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:36.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:36.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:36.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:36.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:36.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:36.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:36.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:36.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:37.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:36.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:37.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:36.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:37.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:36.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:37.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:37.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:37.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:37.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:37.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:37.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:37.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:37.333+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:38.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:37.747+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5524 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:37.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:37.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:37.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:38.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:38.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:38.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:38.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:38.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:38.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:38.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:38.301+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:38.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:38.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:38.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:39.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:38.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:39.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:38.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:39.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:38.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:39.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:39.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:39.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:39.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:39.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:39.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:39.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:39.336+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:39.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:39.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:39.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:40.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:40.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:40.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:40.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:40.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:40.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:40.386+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:40.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:40.386+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:41.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:40.922+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:41.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:40.922+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:41.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:40.922+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:41.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:41.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:41.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:41.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:41.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:41.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:41.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:41.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:41.403+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:42.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:41.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:42.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:41.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:42.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:41.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:42.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:42.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:42.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:42.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:42.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:42.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:42.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:42.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:42.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:42.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:42.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:42.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:42.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:42.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:43.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:42.748+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5605 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:42.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:42.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:42.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:43.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:43.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:43.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:43.365+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:43.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:43.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:43.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:43.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:43.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:43.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:43.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:43.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:43.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:44.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:44.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:44.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:44.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:44.835 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:44.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:44.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:44.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:44.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:44.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:44.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:44.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:44.836 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:44.342+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:45.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:44.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:45.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:44.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:45.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:44.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:45.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:45.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:45.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:45.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:45.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:45.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:45.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:45.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:45.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:45.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:45.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:45.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:46.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:45.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:46.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:45.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:46.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:45.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:46.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:46.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:46.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:46.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:46.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:46.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:46.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:46.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:46.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:46.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:46.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:46.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:46.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:46.340+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:47.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:46.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:47.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:46.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:47.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:46.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:47.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:47.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:47.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:47.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:47.744 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:47.381+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:47.744 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:47.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:47.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:47.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:47.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:47.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:47.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:47.383+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:48.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:47.749+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5689 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:47.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:47.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:47.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:48.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:48.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:48.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:48.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:48.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:48.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:48.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:48.338+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:49.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:48.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:49.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:48.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:49.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:48.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:49.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:49.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:49.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:49.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:49.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:49.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:49.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:49.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:49.311+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:50.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:49.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:50.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:49.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:50.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:49.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:50.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:50.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:50.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:50.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:50.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:50.378+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:50.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:50.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:50.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:50.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:50.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:50.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:50.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:50.309+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:51.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:50.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:51.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:50.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:51.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:50.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:51.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:51.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:51.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:51.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:51.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:51.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:51.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:51.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:51.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:51.776 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:23:51.780+0000 7f7f22242700 0 monclient(hunting): authenticate timed out after 300 2023-12-19T19:23:51.776 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-19T19:23:52.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:51.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:52.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:51.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:52.048 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:51.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:52.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:52.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:52.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:52.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:52.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:52.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:52.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:52.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:52.270+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:52.516 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2023-12-19T19:23:52.804 INFO:teuthology.orchestra.run.smithi055.stderr:Inferring config /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/mon.smithi055/config 2023-12-19T19:23:53.010 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:52.750+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5771 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:53.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:52.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:53.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:52.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:53.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:52.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:53.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:53.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:53.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:53.318+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:53.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:53.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:53.576 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:53.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:54.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:53.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:54.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:53.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:54.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:53.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:54.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:54.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:54.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:54.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:54.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:54.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:54.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:54.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:54.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:54.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:54.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:54.362+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:54.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:54.362+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:55.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:54.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:55.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:54.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:55.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:54.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:55.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:55.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:55.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:55.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:55.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:55.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:55.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:55.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:55.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:56.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:55.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:56.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:55.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:56.279 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:55.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:56.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:56.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:56.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:56.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:56.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:56.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:56.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:56.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:56.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:56.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:56.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:56.327+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:56.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:56.327+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:57.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:56.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:57.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:56.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:56.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:57.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:57.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:57.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:57.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:57.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:57.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:57.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:57.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:57.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:57.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:57.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:57.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:57.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:57.330+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:58.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:23:57.751+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5856 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:23:58.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:57.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:58.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:57.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:58.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:57.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:58.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:58.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:58.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:58.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:58.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:58.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:58.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:58.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:58.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:58.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:58.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:58.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:58.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:58.318+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:23:59.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:58.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:59.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:58.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:59.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:58.914+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:23:59.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:59.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:59.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:59.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:59.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:23:59.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:23:59.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:59.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:59.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:59.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:59.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:59.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:23:59.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:23:59.361+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:00.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:59.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:35.662858+0000 front 2023-12-19T19:13:35.662843+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:24:00.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:59.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:35.662976+0000 front 2023-12-19T19:13:35.662790+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:24:00.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:23:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:23:59.878+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:35.662824+0000 front 2023-12-19T19:13:35.662781+0000 (oldest deadline 2023-12-19T19:13:59.762660+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:00.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:00.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:00.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:00.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:00.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:00.392+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:00.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:00.392+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:01.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:36.591138+0000 front 2023-12-19T19:13:36.591153+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:01.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:36.591199+0000 front 2023-12-19T19:13:36.591283+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:01.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:36.591297+0000 front 2023-12-19T19:13:36.591317+0000 (oldest deadline 2023-12-19T19:13:57.691089+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:01.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:01.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:01.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:01.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:01.420+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:02.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:02.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:02.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:02.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:02.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:02.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:02.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:02.427+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:03.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:02.752+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 5937 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:03.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:03.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:03.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:03.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:03.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:03.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:03.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:03.412+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:04.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:04.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:04.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:04.370+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:05.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:05.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:05.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:05.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:05.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:05.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:05.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:05.400+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:06.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:06.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:06.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:06.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:06.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:06.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:06.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:06.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:07.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:07.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 since back 2023-12-19T19:13:34.817197+0000 front 2023-12-19T19:13:34.817228+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:07.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:07.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 since back 2023-12-19T19:13:34.817368+0000 front 2023-12-19T19:13:34.817316+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:07.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:07.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 since back 2023-12-19T19:13:34.817263+0000 front 2023-12-19T19:13:34.817407+0000 (oldest deadline 2023-12-19T19:14:00.117115+0000) 2023-12-19T19:24:07.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:07.352+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:08.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:07.754+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6020 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:08.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:08.380+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:09.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:09.376+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:10.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:10.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:11.657 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:11.398+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:12.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:12.362+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:13.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:12.755+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6103 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:13.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:13.410+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:14.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:14.366+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:15.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:15.409+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:16.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:16.425+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:17.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:17.394+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:18.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:17.756+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6187 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:18.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:18.383+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:19.342+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:20.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:20.310+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:21.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:21.337+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:22.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:22.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:23.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:22.759+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6269 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:23.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:23.387+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:24.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:23.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:24.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:23.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:24.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:23.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:24.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:24.394+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:24.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:24.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:24.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:24.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:24.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:24.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:25.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:24.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:25.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:24.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:25.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:24.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:25.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:25.441+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:25.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:25.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:25.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:25.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:25.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:25.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:26.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:25.853+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:26.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:25.853+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:26.298 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:25.853+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:26.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:26.396+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:26.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:26.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:26.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:27.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:26.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:27.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:26.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:27.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:26.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:27.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:27.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:27.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:27.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:27.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:27.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:27.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:27.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:28.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:27.760+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6352 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:28.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:27.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:28.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:27.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:28.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:27.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:28.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:28.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:28.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:28.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:28.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:28.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:28.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:28.464+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:29.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:28.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:29.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:28.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:29.274 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:28.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:29.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:29.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:29.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:29.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:29.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:29.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:29.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:29.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:30.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:29.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:30.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:29.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:30.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:29.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:30.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:30.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:30.290+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:30.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:30.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:30.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:30.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:30.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:30.411+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:31.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:30.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:31.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:30.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:31.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:30.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:31.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:31.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:31.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:31.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:31.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:31.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:31.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:31.546 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:31.367+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:32.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:31.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:32.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:31.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:32.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:31.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:32.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:32.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:32.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:32.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:32.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:32.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:32.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:32.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:32.403+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:33.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:32.761+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6435 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:32.791+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:32.791+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:33.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:32.791+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:33.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:33.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:33.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:33.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:33.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:33.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:33.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:33.453+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:33.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:33.453+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:33.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:33.453+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:33.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:33.453+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:34.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:33.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:34.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:33.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:34.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:33.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:34.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:34.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:34.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:34.470+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:34.470+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:34.470+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:34.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:34.470+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:35.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:34.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:35.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:34.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:35.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:34.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:35.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:35.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:35.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:35.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:35.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:35.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:35.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:35.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:35.451+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:35.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:35.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:36.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:35.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:36.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:36.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:36.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:36.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:36.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:36.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:36.781 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:36.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:37.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:36.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:37.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:36.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:37.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:36.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:37.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:37.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:37.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:37.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:37.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:37.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:37.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:37.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:37.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:37.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:37.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:37.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:37.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:37.463+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:38.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:37.763+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6519 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:37.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:37.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:38.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:37.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:38.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:38.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:38.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:38.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:38.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:38.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:38.403+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:38.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:38.403+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:39.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:38.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:39.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:38.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:39.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:38.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:39.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:39.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:39.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:39.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:39.415+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:39.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:39.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:39.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:39.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:39.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:40.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:39.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:40.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:40.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:40.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:40.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:40.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:40.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:40.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:40.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:40.492+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:40.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:40.492+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:40.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:40.492+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:41.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:40.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:41.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:40.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:41.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:40.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:41.659 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:41.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:41.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:41.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:41.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:41.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:41.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:41.660 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:41.376+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:42.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:41.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:42.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:41.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:42.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:41.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:42.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:42.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:42.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:42.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:42.727 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:42.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:42.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:42.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:42.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:42.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:42.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:42.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:42.728 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:42.335+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:43.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:42.764+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6601 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:42.860+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:42.860+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:43.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:42.860+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:43.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:43.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:43.579+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:43.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:43.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:43.336+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:43.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:43.336+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:43.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:43.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:43.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:44.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:44.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:44.536+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:44.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:44.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:44.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:44.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:44.369+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:45.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:44.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:45.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:44.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:45.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:44.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:45.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:45.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:45.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:45.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:45.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:45.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:45.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:45.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:45.350+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:46.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:45.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:46.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:45.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:46.296 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:45.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:46.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:46.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:46.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:46.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:46.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:46.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:46.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:46.318+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:46.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:46.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:46.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:46.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:46.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:46.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:46.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:47.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:46.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:47.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:46.882+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:47.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:47.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:47.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:47.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:47.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:47.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:47.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:47.324+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:48.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:47.765+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6684 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:47.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:47.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:48.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:47.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:48.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:48.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:48.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:48.530+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:48.284+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:48.284+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:48.284+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:48.284+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:49.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:48.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:49.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:48.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:49.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:48.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:49.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:49.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:49.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:49.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:49.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:49.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:49.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:49.245+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:50.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:49.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:50.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:49.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:50.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:49.568+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:50.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:49.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:50.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:49.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:50.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:49.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:50.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:50.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:50.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:50.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:50.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:50.253+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:50.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:50.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:50.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:50.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:50.809 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:50.562+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:51.267 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:50.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.267 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:50.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.267 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:50.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:51.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:51.534 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:51.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:51.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:51.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:51.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:51.273+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:51.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:51.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:51.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:51.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:51.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:51.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:51.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:51.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:51.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:51.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:52.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:52.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:52.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:52.283+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:52.283+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:52.283+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:52.283+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:53.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:52.767+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6767 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:53.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:52.739+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:53.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:52.739+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:53.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:52.739+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:53.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:53.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:53.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:53.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:53.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:53.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:53.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:53.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:54.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:53.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:53.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:53.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:53.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:54.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:53.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:54.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:53.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:54.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:54.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:54.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:54.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:54.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:54.198+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:54.524 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:54.198+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:54.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:54.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:54.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:54.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:54.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:54.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:54.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:54.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:54.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:54.529+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:55.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:55.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:55.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:55.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:55.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:55.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:55.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:55.177+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:55.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:55.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:55.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:55.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:55.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:55.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:56.157+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:56.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:56.157+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:56.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:56.157+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:56.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:56.157+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:56.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:56.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:56.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:56.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:56.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:56.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:56.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:56.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:56.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:56.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:57.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:57.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:57.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:57.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:57.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:57.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:57.469 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:57.168+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:57.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:57.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:57.763 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:57.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:57.763 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:57.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:57.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:57.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:57.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:57.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:57.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:57.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:58.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:24:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:24:57.768+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6851 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:24:58.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:58.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:58.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:58.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:58.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:58.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:58.428 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:58.181+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:58.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:58.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:58.434+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:58.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:58.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:58.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:58.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:59.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:59.153+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:59.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:59.153+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:59.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:59.154+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:24:59.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:24:59.154+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:24:59.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:59.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:59.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:59.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:59.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:24:59.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:24:59.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:59.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:59.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:59.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:24:59.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:24:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:24:59.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:00.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:00.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:00.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:00.470 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:00.174+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:00.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:00.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:00.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:00.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:00.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:00.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:00.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:00.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:00.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:00.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:01.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:01.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:01.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:01.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:01.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:01.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:01.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:01.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:01.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:01.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:01.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:01.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:01.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:01.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:01.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:02.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:02.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:02.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:02.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:02.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:02.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:02.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:02.163+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:02.763 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:02.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:02.764 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:02.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:02.764 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:02.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:02.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:02.482+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:02.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:02.482+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:02.764 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:02.482+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:03.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:02.769+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 6933 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:03.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:03.118+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:03.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:03.118+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:03.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:03.118+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:03.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:03.119+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:03.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:03.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:03.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:03.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:03.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:03.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:03.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:03.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:03.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:03.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:03.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:03.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:04.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:04.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:04.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:04.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:04.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:04.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:04.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:04.084+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:04.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:04.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:04.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:04.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:04.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:04.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:04.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:05.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:05.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:05.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:05.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:05.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:05.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:05.452 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:05.127+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:05.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:05.457+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:05.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:05.457+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:05.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:05.457+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:05.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:05.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:05.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:05.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:05.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:05.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:06.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:06.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:06.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:06.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:06.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:06.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:06.448 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:06.127+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:06.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:06.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:06.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:06.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:06.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:06.454+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:06.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:06.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:06.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:07.407 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:07.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:07.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:07.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:07.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:07.107+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:07.408 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:07.107+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:07.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:07.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:07.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:07.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:07.765 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:07.413+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:07.765 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:07.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:07.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:07.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:07.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:07.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:08.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:07.771+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7016 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:08.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:08.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:08.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:08.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:08.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:08.084+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:08.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:08.085+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:08.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:08.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:08.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:08.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:08.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:08.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:09.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:09.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:09.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:09.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:09.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:09.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:09.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:09.053+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:09.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:09.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:09.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:09.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:09.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:09.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:09.543+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:09.543+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:09.543+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:10.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:10.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:10.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:10.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.005+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:10.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:10.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:10.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:10.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:10.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:10.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:10.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:10.524+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:10.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:10.524+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:10.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:10.524+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:11.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:11.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:11.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:11.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:10.981+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:11.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:11.456+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:11.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:11.456+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:11.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:11.456+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:11.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:11.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:11.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:11.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:11.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:11.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:12.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:11.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:12.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:11.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:12.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:11.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:12.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:11.986+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:12.766 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:12.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:12.767 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:12.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:12.767 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:12.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:12.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:12.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:12.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:12.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:12.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:12.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:13.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:12.772+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7099 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:13.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:13.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:13.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:13.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:13.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:13.009+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:13.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:13.009+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:13.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:13.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:13.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:13.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:13.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:13.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:13.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:14.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:14.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:14.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:14.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:14.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:14.040+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:14.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:14.040+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:14.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:14.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:14.495+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:14.420+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:14.420+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:14.420+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:15.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:15.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:15.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:15.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:15.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:15.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:15.440 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:15.071+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:15.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:15.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:15.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:15.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:15.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:15.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:15.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:15.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:15.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:15.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:15.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:15.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:16.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:16.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:16.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:16.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:16.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:16.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:16.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:16.108+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:16.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:16.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:16.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:16.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:16.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:16.453+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:16.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:16.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:16.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:16.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:16.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:16.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:17.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:17.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:17.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:17.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:17.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:17.099+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:17.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:17.099+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:17.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:17.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:17.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:17.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:17.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:17.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:17.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:18.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:17.774+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7183 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:18.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:18.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:18.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:18.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:18.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:18.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:18.453 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:18.120+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:18.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:18.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:18.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:18.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:18.491+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:18.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:18.491+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:18.796 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:18.491+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:19.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:19.104+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:19.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:19.104+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:19.405 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:19.104+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:19.406 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:19.104+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:19.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:19.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:19.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:19.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:19.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:19.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:19.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:19.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:19.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:19.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:19.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:19.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:20.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:20.153+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:20.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:20.153+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:20.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:20.153+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:20.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:20.153+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:20.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:20.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:20.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:20.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:20.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:20.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:20.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:20.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:20.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:21.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:21.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:21.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:21.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:21.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:21.120+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:21.411 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:21.120+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:21.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:21.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:21.663 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:21.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:21.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:21.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:21.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:21.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:21.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:21.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:22.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:22.154+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:22.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:22.154+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:22.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:22.154+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:22.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:22.154+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:22.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:22.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:22.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:22.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:22.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:22.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:22.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:22.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:22.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:22.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:22.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:22.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:23.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:22.775+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7265 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:23.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:23.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:23.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:23.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:23.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:23.146+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:23.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:23.146+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:23.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:23.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:23.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:23.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:23.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:23.452+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:23.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:23.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:23.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:23.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:23.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:23.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:24.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:24.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:24.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:24.408+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:24.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:24.170+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:24.170+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:24.170+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:24.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:24.171+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:24.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:24.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:24.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:25.507 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:25.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:25.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:25.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:25.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:25.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:25.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:25.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:25.199+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:25.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:25.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:25.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:25.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:26.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:26.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:26.460 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:26.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:26.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:26.163+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:26.461 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:26.163+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:26.475+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:26.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:26.475+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:26.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:26.475+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:26.467+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:26.467+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:26.467+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:27.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:27.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:27.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:27.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:27.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:27.181+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:27.456 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:27.182+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:27.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:27.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:27.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:27.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:27.462+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:27.776+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7348 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:28.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:28.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:28.424+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:28.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:28.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:28.217+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:28.498 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:28.217+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:28.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:28.504+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:28.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:28.504+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:28.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:28.504+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:29.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:29.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:29.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:29.437+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:29.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:29.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:29.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:29.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:29.212+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:29.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:29.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:29.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:30.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:30.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:30.212+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:30.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:30.212+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:30.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:30.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:30.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:30.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:30.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:30.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:30.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:30.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:30.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:31.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:31.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:31.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:31.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:31.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:31.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:31.493 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:31.178+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:31.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:31.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:31.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:31.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:31.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:31.499+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:31.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:31.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:31.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:31.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:31.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:31.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:32.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:32.180+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:32.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:32.180+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:32.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:32.180+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:32.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:32.180+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:32.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:32.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:32.450+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:32.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:32.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:32.772 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:32.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:33.045 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:32.778+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7431 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:33.454 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:33.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:33.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:33.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:33.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:33.155+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:33.455 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:33.155+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:33.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:33.461+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:33.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:33.461+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:33.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:33.461+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:33.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:33.491+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:33.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:33.491+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:33.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:33.491+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:34.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:34.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:34.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:34.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:34.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:34.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:34.477 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:34.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:34.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:34.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:34.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:34.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:34.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:34.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:34.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:34.505+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:34.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:34.505+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:34.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:34.505+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:35.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:35.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:35.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:35.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:35.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:35.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:35.295 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:35.169+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:35.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:35.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:35.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:35.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:35.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:35.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:35.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:35.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:35.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:35.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:35.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:36.446 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:36.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:36.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:36.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:36.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:36.182+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:36.447 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:36.182+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:36.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:36.535+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:36.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:36.535+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:36.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:36.535+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:36.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:36.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:36.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:36.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:36.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:36.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:37.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:37.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:37.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:37.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:37.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:37.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:37.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:37.168+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:37.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:37.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:37.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:37.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:37.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:37.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:37.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:37.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:37.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:37.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:37.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:37.451+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:38.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:37.779+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7515 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:38.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:38.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:38.421 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:38.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:38.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:38.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:38.422 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:38.177+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:38.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:38.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:38.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:38.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:38.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:38.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:38.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:38.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:38.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:38.533+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:39.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:39.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:39.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:39.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:39.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:39.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:39.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:39.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:39.204+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:39.509+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:39.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:39.509+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:39.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:39.509+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:40.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:40.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:40.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:40.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:40.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:40.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:40.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:40.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:40.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:40.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:40.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:40.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:40.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:40.229+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:40.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:40.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:40.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:40.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:40.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:40.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:41.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:41.214+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:41.536 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:41.214+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:41.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:41.214+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:41.537 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:41.214+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:41.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:41.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:41.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:41.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:41.537 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:41.348+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:41.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:41.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:41.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:41.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:41.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:41.543+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:42.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:42.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:42.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:42.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:42.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:42.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:42.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:42.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:42.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:42.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:42.254+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:42.899 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:42.781+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7597 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:42.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:42.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:42.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:42.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:42.899 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:42.574+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:43.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:43.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:43.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:43.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:43.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:43.390+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:43.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:43.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:43.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:43.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:43.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:43.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:43.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:43.281+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:44.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:43.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:44.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:43.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:44.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:43.551+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:44.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:44.412+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:44.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:44.412+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:44.531 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:44.412+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:44.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:44.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:44.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:44.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:44.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:44.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:44.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:44.240+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:44.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:44.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:44.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:44.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:44.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:44.537+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:45.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:45.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:45.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:45.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:45.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:45.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:45.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:45.269+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:45.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:45.269+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:45.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:45.269+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:45.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:45.269+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:46.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:45.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:46.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:45.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:46.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:45.554+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:46.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:46.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:46.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:46.221+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:46.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:46.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:46.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:46.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:46.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:46.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:47.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:46.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:47.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:46.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:47.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:46.576+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:47.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:47.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:47.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:47.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:47.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:47.253+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:47.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:47.253+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:47.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:47.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:47.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:47.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:47.776 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:47.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:48.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:47.782+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7680 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:48.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:48.375+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:48.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:48.375+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:48.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:48.375+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:48.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:48.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:48.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:48.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:48.266+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:49.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:48.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:49.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:48.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:49.045 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:48.556+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:49.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:49.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:49.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:49.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:49.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:49.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:49.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:49.228+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:49.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:49.228+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:49.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:49.228+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:49.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:49.228+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:49.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:49.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:49.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:49.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:49.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:49.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:50.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:50.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:50.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:50.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:50.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:50.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:50.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:50.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:50.480+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:50.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:50.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:50.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:50.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:50.243+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:51.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:51.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:51.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:51.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:51.484 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:51.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:51.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:51.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:51.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:51.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:51.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:51.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:51.485 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:51.224+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:51.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:51.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:51.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:51.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:51.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:51.490+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:52.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:52.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:52.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:52.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:52.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:52.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:52.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:52.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:52.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:52.264+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:52.264+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:52.264+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:52.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:52.264+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:53.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:52.784+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7763 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:53.383 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:53.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:53.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:53.226+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:53.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:53.226+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:53.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:53.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:53.517+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:53.395+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:53.395+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:53.395+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:54.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:54.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:54.240+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:54.240+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:54.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:54.507 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:54.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:54.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:54.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:54.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:54.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:54.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:54.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:54.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:54.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:55.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:55.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:55.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:55.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:55.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:55.513+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:55.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:55.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:55.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:55.287+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:55.369+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:55.369+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:55.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:55.369+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:56.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:56.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:56.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:56.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:56.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:56.504+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:56.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:56.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:56.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:56.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:56.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:56.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:56.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:56.296+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:57.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:57.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:57.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:57.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:57.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:57.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:57.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:57.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:57.479+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:57.268+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:57.268+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:57.268+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:57.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:57.269+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:58.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:25:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:25:57.785+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7847 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:25:58.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:58.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:58.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:58.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:58.480 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:58.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:58.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:58.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:58.224+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:58.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:58.224+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:58.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:58.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:58.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:58.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:58.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:58.487+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:59.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:59.186+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:59.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:59.186+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:59.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:59.186+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:25:59.439 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:25:59.186+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:25:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:59.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:59.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:25:59.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:25:59.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:59.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:59.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:59.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:25:59.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:25:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:25:59.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:00.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:00.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:00.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:00.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:00.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:00.148+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:00.442 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:00.148+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:00.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:00.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:00.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:00.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:00.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:00.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:00.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:00.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:01.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:01.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:01.444 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:01.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:01.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:01.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:01.445 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:01.193+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:01.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:01.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:01.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:01.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:01.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:01.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:01.451+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:02.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:02.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:02.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:02.210+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:02.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:02.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:02.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:02.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:02.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:02.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:02.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:02.905 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:02.787+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 7923 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:03.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:03.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:03.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:03.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:03.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:03.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:03.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:03.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:03.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:03.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:03.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:03.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:03.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:03.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:03.248+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:04.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:04.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:04.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:04.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:04.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:04.405+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:04.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:04.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:04.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:04.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:04.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:04.281+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:04.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:04.281+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:05.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:05.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:05.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:05.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:05.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:05.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:05.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:05.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:05.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:05.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:05.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:05.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:05.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:05.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:05.279+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:06.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:06.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:06.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:06.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:06.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:06.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:06.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:06.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:06.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:06.311+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:06.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:06.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:06.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:06.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:07.696 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:07.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:07.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:07.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:07.697 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:07.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:07.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:07.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:07.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:07.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:07.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:07.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:07.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:07.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:07.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:07.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:07.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:07.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:07.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:07.337+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:08.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:07.788+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8000 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:08.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:08.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:08.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:08.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:08.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:08.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:08.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:08.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:08.333+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:08.333+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:08.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:08.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:08.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:08.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:09.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:09.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:09.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:09.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:09.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:09.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:09.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:09.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:09.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:09.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:09.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:09.337+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:09.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:09.337+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:10.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:10.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:10.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:10.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:10.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:10.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:10.565+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:10.565+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:10.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:10.565+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:10.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:10.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:10.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:10.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:10.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:10.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:10.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:10.373+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:11.664 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:11.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:11.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:11.493+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:11.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:11.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:11.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:11.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:11.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:11.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:11.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:11.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:11.358+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:12.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:12.533+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:12.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:12.533+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:12.784 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:12.533+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:12.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:12.447+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:12.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:12.447+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:12.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:12.447+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:12.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:12.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:12.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:12.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:12.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:12.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:12.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:12.357+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:13.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:12.790+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8074 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:13.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:13.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:13.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:13.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:13.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:13.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:13.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:13.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:13.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:13.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:13.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:13.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:13.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:13.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:13.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:13.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:13.331+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:14.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:14.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:14.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:14.484+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:14.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:14.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:14.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:14.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:14.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:14.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:14.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:14.331+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:15.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:15.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:15.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:15.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:15.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:15.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:15.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:15.579 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:15.320+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:16.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:15.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:16.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:15.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:16.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:15.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:16.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:16.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:16.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:16.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:16.592 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:16.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:16.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:16.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:16.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:16.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:16.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:16.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:16.593 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:16.344+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:17.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:16.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:16.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:16.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:17.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:17.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:17.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:17.459+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:17.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:17.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:17.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:17.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:17.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:17.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:17.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:17.365+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:18.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:17.792+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8149 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:18.359+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:18.360+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:18.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:18.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:18.430+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:18.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:18.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:18.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:18.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:19.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:19.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:19.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:19.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:19.612 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:19.448+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:19.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:19.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:19.612 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:19.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:19.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:19.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:19.613 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:19.330+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:20.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:19.619+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:20.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:19.619+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:20.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:19.619+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:20.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:20.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:20.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:20.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:20.645 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:20.444+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:20.645 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:20.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:20.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:20.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:20.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:20.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:20.646 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:20.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:21.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:20.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:21.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:20.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:21.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:20.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:21.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:21.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:21.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:21.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:21.665 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:21.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:21.665 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:21.375+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:21.375+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:21.375+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:21.375+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:21.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:21.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:21.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:21.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:22.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:22.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:22.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:22.332+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:22.332+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:22.332+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:22.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:22.332+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:22.911 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:22.793+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8222 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:22.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:22.602+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:22.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:22.602+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:22.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:22.602+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:23.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:23.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:23.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:23.383+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:23.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:23.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:23.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:23.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:24.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:23.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:24.070 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:23.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:24.070 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:23.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:24.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:24.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:24.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:24.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:24.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:24.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:24.376 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:24.354+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:24.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:24.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:24.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:24.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:24.647 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:24.383+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:25.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:24.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:25.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:24.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:25.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:24.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:25.620 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:25.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:25.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:25.361+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:25.361+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:25.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:25.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:25.621 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:25.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:26.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:26.593+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:26.593+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:26.593+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:26.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:26.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:26.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:26.346+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:26.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:26.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:26.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:26.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:27.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:27.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:27.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:27.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:27.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:27.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:27.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:27.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:27.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:27.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:27.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:27.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:27.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:27.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:27.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:28.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:27.795+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8302 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:28.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:28.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:28.361+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:28.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:28.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:28.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:28.609 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:28.334+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:29.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:28.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:29.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:28.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:29.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:28.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:29.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:29.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:29.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:29.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:29.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:29.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:29.607 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:29.315+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:30.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:29.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:30.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:29.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:30.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:29.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:30.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:30.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:30.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:30.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:30.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:30.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:30.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:30.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:30.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:30.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:30.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:30.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:30.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:30.282+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:31.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:30.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:31.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:30.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:31.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:30.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:31.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:31.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:31.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:31.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:31.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:31.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:31.551 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:31.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:32.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:31.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:32.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:31.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:32.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:31.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:32.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:32.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:32.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:32.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:32.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:32.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:32.596 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:32.311+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:32.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:32.354+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:32.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:32.354+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:32.597 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:32.354+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:32.915 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:32.796+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8385 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:32.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:32.603+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:32.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:32.603+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:32.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:32.603+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:33.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:33.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:33.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:33.291+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:33.380+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:33.380+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:33.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:33.380+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:34.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:33.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:34.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:33.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:34.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:33.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:34.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:34.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:34.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:34.309+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:34.309+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:34.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:34.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:34.643 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:34.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:35.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:34.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:35.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:34.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:35.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:34.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:35.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:35.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:35.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:35.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:35.636 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:35.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:35.637 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:35.350+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:35.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:35.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:35.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:35.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:35.637 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:35.363+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:36.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:35.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:36.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:35.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:36.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:35.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:36.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:36.310+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:36.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:36.310+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:36.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:36.310+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:36.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:36.310+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:36.656 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:36.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:36.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:36.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:36.657 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:36.343+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:37.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:36.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:37.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:36.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:37.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:36.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:37.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:37.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:37.311+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:37.311+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:37.632 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:37.633 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:37.335+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:37.917 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:37.798+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8469 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:37.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:37.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:37.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:37.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:37.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:37.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:38.278+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:38.278+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:38.278+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:38.278+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:38.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:38.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:38.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:38.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:39.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:38.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:39.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:38.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:39.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:38.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:39.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:39.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:39.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:39.247+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:39.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:39.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:39.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:39.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:39.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:39.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:40.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:39.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:40.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:39.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:40.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:39.668+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:40.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:40.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:40.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:40.273+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:40.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:40.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:40.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:40.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:41.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:40.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:41.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:40.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:41.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:40.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:41.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:41.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:41.319+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:41.319+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:41.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:41.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:41.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:41.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:42.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:41.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:42.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:41.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:42.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:41.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:42.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:42.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:42.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:42.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:42.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:42.353+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:42.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:42.353+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:42.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:42.353+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:42.535 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:42.353+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:42.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:42.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:42.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:42.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:42.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:42.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:43.044 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:42.799+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8551 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:43.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:43.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:43.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:43.315+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:43.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:43.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:43.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:43.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:43.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:43.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:43.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:44.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:44.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:44.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:44.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:44.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:44.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:44.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:44.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:44.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:44.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:44.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:44.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:44.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:44.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:44.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:45.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:45.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:45.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:45.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:45.338+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:45.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:45.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:45.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:45.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:45.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:45.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:45.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:46.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:46.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:46.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:46.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:46.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:46.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:46.520 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:46.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:46.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:46.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:46.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:46.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:46.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:46.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:47.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:47.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:47.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:47.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:47.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:47.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:47.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:47.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:47.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:47.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:47.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:47.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:47.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:47.404+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:48.082 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:47.801+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8634 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:48.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:48.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:48.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:48.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:48.400 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:48.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:48.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:48.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:48.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:48.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:48.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:48.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:48.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:48.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:48.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:48.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:48.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:48.407+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:49.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:49.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:49.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:49.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:49.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:49.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:49.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:49.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:49.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:49.403+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:49.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:49.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:49.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:50.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:50.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:50.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:50.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:50.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:50.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:50.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:50.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:50.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:50.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:50.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:50.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:50.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:50.398+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:51.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:51.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:51.392 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:51.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:51.393 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:51.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:51.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:51.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:51.666 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:51.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:51.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:51.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:51.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:51.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:51.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:51.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:51.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:51.399+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:52.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:52.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:52.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:52.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:52.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:52.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:52.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:52.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:52.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:52.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:52.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:52.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:52.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:52.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:53.200 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:52.802+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8717 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:53.206+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:53.206+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:53.206+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:53.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:53.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:53.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:53.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:53.331+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:53.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:53.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:53.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:53.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:54.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:54.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:54.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:54.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:54.479 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:54.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:54.479 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:54.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:54.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:54.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:54.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:54.342+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:54.480 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:54.342+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:54.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:54.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:54.809 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:54.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:54.810 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:54.485+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:55.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:55.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:55.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:55.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:55.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:55.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:55.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:55.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:55.372+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:55.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:55.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:55.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:55.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:55.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:55.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:56.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:56.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:56.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:56.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:56.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:56.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:56.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:56.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:56.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:56.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:56.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:56.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:56.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:56.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:57.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:57.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:57.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:57.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:57.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:57.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:57.441 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:57.325+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:57.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:57.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:57.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:58.155 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:26:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:26:57.804+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8801 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:26:58.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:58.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:58.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:58.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:58.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:58.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:58.426 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:58.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:58.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:58.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:58.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:58.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:58.427 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:58.302+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:58.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:58.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:58.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:58.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:58.433+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:59.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:59.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:26:59.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:59.257+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:59.257+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:59.257+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:26:59.476 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:26:59.257+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:26:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:59.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:59.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:26:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:26:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:26:59.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:00.136+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:00.136+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:00.136+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:00.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:00.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:00.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:00.488 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:00.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:00.495+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:00.495+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:00.495+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:01.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:01.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:01.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:01.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:01.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:01.516 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:01.248+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:01.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:01.248+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:01.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:01.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:01.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:02.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:02.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:02.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:02.210+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:02.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:02.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:02.544 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:02.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:02.813 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:02.806+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8883 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:02.814 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:02.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:02.814 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:02.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:02.814 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:02.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:03.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:03.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:03.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:03.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:03.527 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:03.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:03.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:03.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:03.527 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:03.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:03.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:03.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:03.528 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:03.252+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:03.534+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:03.534+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:03.534+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:04.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:04.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:04.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:04.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:04.254+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:05.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:05.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:05.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:05.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:05.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:05.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:05.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:05.296+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:05.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:05.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:05.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:05.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:05.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:05.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:05.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:05.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:05.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:05.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:05.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:05.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:06.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:06.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:06.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:06.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:06.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:06.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:06.329+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:06.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:06.329+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:06.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:06.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:06.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:07.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:07.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:07.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:07.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:07.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:07.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:07.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:07.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:07.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:07.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:07.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:07.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:07.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:07.299+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:07.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:07.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:07.796 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:07.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:07.797 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:07.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:08.111 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:07.807+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 8966 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:08.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:08.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:08.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:08.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:08.256+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:08.256+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:08.256+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:08.484 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:08.256+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:08.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:08.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:08.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:08.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:08.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:09.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:09.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:09.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:09.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:09.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:09.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:09.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:09.261+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:09.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:09.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:09.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:09.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:09.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:09.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:10.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:10.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:10.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:10.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:10.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:10.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:10.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:10.243+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:10.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:10.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:10.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:10.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:11.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:11.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:11.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:11.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:11.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:11.291+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:11.451 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:11.291+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:11.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:11.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:11.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:12.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:12.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:12.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:12.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:12.449 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:12.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:12.450 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:12.328+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:12.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:12.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:12.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:12.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:12.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:12.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:12.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:12.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:12.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:12.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:13.127 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:12.809+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9049 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:13.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:13.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:13.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:13.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:13.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:13.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:13.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:13.338+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:13.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:13.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:13.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:14.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:14.111+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:14.111+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:14.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:14.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:14.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:14.486 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:14.357+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:14.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:14.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:14.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:14.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:14.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:15.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:15.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:15.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:15.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:15.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:15.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:15.503+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:15.503+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:15.503+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:15.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:15.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:15.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:15.397+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:16.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:16.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:16.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:16.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:16.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:16.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:16.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:16.430+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:16.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:16.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:16.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:16.552+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:17.198+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:17.198+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:17.198+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:17.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:17.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:17.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:17.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:17.434+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:17.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:17.507+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:17.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:17.507+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:17.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:17.507+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:18.173 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:17.811+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9133 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:18.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:18.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:18.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:18.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:18.445 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:18.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:18.537+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:18.537+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:18.537+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:18.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:18.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:18.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:18.451+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:19.471 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:19.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:19.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:19.472 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:19.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:19.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:19.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:19.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:19.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:19.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:19.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:19.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:20.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:20.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:20.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:20.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:20.512 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:20.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:20.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:20.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:20.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:20.520+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:20.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:20.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:20.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:21.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:21.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:21.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:21.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:21.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:21.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:21.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:21.553+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:21.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:21.553+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:21.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:21.553+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:21.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:21.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:21.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:21.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:21.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:21.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:21.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:21.540+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:22.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:22.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:22.470 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:22.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:22.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:22.540+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:22.540+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:22.540+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:22.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:22.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:22.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:22.502+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:23.114 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:22.812+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9215 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:23.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:23.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:23.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:23.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:23.461 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:23.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:23.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:23.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:23.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:23.469+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:23.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:23.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:23.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:23.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:23.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:23.499+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:24.124+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:24.124+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:24.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:24.124+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:24.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:24.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:24.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:24.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:24.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:24.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:24.448+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:25.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:25.095+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:25.439 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:25.095+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:25.440 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:25.095+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:25.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:25.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:25.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:25.468+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:25.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:25.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:25.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:26.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:26.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:26.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:26.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:26.420 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:26.086+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:26.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:26.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:26.512+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:26.512+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:26.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:26.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:26.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:27.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:27.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:27.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:27.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:27.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:27.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:27.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:27.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:27.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:27.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:27.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:27.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:27.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:28.099 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:27.814+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9298 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:28.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:28.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:28.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:28.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:28.399+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:28.399+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:28.399+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:28.526+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:28.526+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:28.526+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:28.526+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:29.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:29.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:29.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:29.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:29.424 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:29.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:29.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:29.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:29.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:29.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:29.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:29.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:29.519+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:30.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:30.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:30.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:30.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:30.468 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:30.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:30.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:30.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:30.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:30.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:30.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:30.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:30.795 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:30.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:31.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:31.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:31.422 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:31.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:31.423 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:31.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:31.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:31.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:31.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:31.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:31.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:31.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:31.686 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:31.489+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:32.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:32.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:32.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:32.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:32.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:32.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:32.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:32.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:32.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:32.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:32.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:32.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:32.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:32.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:32.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:32.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:32.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:32.488+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:32.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:32.488+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:33.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:32.816+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9381 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:33.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:33.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:33.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:33.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:33.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:33.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:33.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:33.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:33.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:33.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:33.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:33.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:33.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:33.509+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:34.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:34.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:34.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:34.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:34.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:34.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:34.548+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:34.548+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:34.548+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:34.548+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:34.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:34.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:34.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:34.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:35.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:35.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:35.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:35.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:35.429 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:35.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:35.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:35.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:35.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:35.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:35.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:35.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:35.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:35.516+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:36.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:36.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:36.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:36.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:36.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:36.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:36.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:36.428+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:36.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:36.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:36.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:36.560+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:37.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:37.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:37.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:37.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:37.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:37.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:37.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:37.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:37.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:37.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:37.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:37.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:37.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:37.554+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:38.123 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:37.817+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9465 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:38.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:38.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:38.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:38.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:38.408 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:38.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:38.415+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:38.415+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:38.415+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:38.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:38.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:38.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:38.597+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:39.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:39.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:39.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:39.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:39.450 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:39.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:39.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:39.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:39.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:39.632+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:39.632+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:39.632+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:39.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:39.632+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:40.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:40.149+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:40.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:40.149+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:40.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:40.149+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:40.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:40.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:40.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:40.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:40.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:40.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:40.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:40.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:40.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:40.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:40.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:40.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:40.629+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:41.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:41.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:41.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:41.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:41.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:41.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:41.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:41.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:41.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:41.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:41.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:41.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:41.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:41.616+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:42.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:42.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:42.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:42.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:42.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:42.406+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:42.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:42.406+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:42.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:42.406+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:42.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:42.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:42.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:42.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:42.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:42.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:42.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:42.571+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:43.067 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:42.819+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9546 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:43.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:43.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:43.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:43.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:43.355 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:43.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:43.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:43.361+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:43.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:43.361+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:43.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:43.361+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:43.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:43.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:43.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:43.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:43.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:43.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:43.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:43.524+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:44.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:44.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:44.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:44.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:44.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:44.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:44.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:44.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:44.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:44.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:44.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:44.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:44.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:44.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:44.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:44.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:44.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:44.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:44.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:44.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:45.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:45.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:45.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:45.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:45.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:45.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:45.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:45.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:45.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:45.377+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:45.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:45.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:45.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:45.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:45.509+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:46.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:46.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:46.350 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:46.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:46.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:46.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:46.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:46.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:46.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:46.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:46.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:46.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:46.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:46.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:46.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:46.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:46.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:46.479+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:47.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:47.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:47.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:47.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:47.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:47.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:47.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:47.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:47.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:47.432+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:47.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:47.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:47.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:47.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:48.091 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:47.821+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9630 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:48.363 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:48.098+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:48.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:48.098+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:48.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:48.098+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:48.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:48.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:48.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:48.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:48.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:48.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:48.418+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:49.380 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:49.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:49.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:49.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:49.381 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:49.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:49.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:49.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:49.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:49.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:49.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:49.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:49.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:49.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:50.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:50.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:50.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:50.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:50.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:50.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:50.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:50.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:50.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:50.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:50.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:50.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:50.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:50.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:50.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:50.481+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:51.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:51.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:51.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:51.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:51.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:51.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:51.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:51.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:51.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:51.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:51.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:51.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:51.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:51.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:51.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:51.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:51.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:51.517+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:52.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:52.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:52.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:52.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:52.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:52.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:52.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:52.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:52.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:52.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:52.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:52.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:52.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:52.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:52.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:52.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:52.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:53.124 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:52.822+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9712 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:53.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:53.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:53.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:53.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:53.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:53.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:53.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:53.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:53.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:53.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:53.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:53.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:53.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:53.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:53.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:53.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:53.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:53.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:53.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:53.555+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:54.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:54.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:54.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:54.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:54.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:54.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:54.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:54.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:54.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:54.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:54.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:54.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:54.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:54.558+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:55.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:55.075+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:55.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:55.075+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:55.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:55.075+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:55.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:55.441+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:55.441+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:55.441+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:55.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:55.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:55.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:55.563+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:56.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:56.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:56.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:56.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:56.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:56.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:56.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:56.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:56.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:56.541+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:56.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:56.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:56.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:56.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:56.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:56.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:57.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:57.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:57.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:57.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:57.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:57.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:57.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:57.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:57.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:57.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:57.585+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:57.444+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:57.444+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:57.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:57.444+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:58.135 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:27:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:27:57.824+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9797 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:27:58.135 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:58.038+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:58.135 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:58.038+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:58.135 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:58.038+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:58.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:58.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:58.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:58.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:58.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:58.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:58.584+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:27:59.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:59.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:59.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:27:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:27:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:59.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:59.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:27:59.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:27:59.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:59.608+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:59.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:59.608+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:59.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:59.608+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:27:59.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:27:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:27:59.608+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:00.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:00.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:00.449 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:00.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:00.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:00.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:00.456+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:00.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:00.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:00.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:00.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:00.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:00.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:00.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:00.589+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:01.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:01.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:01.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:01.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:01.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:01.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:01.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:01.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:01.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:01.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:01.574+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:01.574+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:01.574+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:01.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:01.574+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:02.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:02.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:02.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:02.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:02.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:02.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:02.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:02.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:02.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:02.559+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:02.559+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:02.559+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:02.559+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:03.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:02.826+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9878 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:03.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:03.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:03.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:03.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:03.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:03.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:03.453+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:03.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:03.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:03.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:03.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:03.580+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:03.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:04.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:04.422+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:04.422+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:04.422+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:04.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:04.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:04.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:04.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:04.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:04.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:04.562+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:05.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:04.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:05.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:04.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:05.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:04.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:05.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:05.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:05.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:05.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:05.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:05.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:05.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:05.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:05.600+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:06.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:05.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:06.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:05.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:06.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:05.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:06.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:06.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:06.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:06.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:06.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:06.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:06.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:06.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:06.602+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:06.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:06.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:07.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:06.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:07.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:07.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:07.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:07.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:07.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:07.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:07.820 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:07.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:07.821 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:07.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:07.821 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:07.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:07.821 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:07.572+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:08.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:07.827+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 9962 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:07.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:08.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:07.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:08.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:07.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:08.354+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:08.354+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:08.354+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:08.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:08.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:08.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:08.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:08.552+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:08.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:08.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:08.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:09.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:09.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:09.357+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:09.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:09.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:09.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:09.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:09.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:09.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:09.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:09.565+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:10.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:09.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:10.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:09.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:10.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:09.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:10.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:10.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:10.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:10.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:10.516+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:10.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:10.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:10.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:10.329+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:10.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:10.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:10.875+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:11.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:11.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:11.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:11.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:11.473+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:12.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:11.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:12.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:11.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:12.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:11.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:12.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:12.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:12.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:12.517+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:12.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:12.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:12.372+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:13.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:12.829+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10044 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:13.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:12.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:13.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:12.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:13.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:12.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:13.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:13.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:13.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:13.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:13.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:13.516+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:13.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:13.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:13.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:13.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:13.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:13.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:14.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:14.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:14.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:14.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:14.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:14.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:14.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:14.530+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:15.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:14.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:15.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:14.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:15.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:14.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:15.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:15.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:15.349+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:15.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:15.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:15.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:15.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:15.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:15.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:15.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:16.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:16.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:16.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:16.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:16.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:16.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:16.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:16.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:16.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:16.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:16.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:17.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:16.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:17.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:16.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:17.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:16.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:17.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:17.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:17.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:17.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:17.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:17.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:17.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:17.447+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:18.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:17.831+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10129 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:18.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:17.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:18.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:17.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:18.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:17.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:18.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:18.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:18.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:18.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:18.339+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:18.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:18.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:18.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:18.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:18.445+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:19.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:18.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:19.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:18.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:19.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:18.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:19.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:19.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:19.362+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:19.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:19.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:19.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:19.433+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:20.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:19.929+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:20.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:19.929+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:20.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:19.929+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:20.342+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:20.342+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:20.342+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:20.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:20.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:20.391+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:20.391+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:20.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:20.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:20.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:21.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:21.309+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:21.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:21.309+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:21.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:21.309+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:21.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:21.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:21.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:21.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:21.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:21.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:21.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:21.433+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:21.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:21.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:21.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:22.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:22.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:22.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:22.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:22.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:22.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:22.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:22.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:22.401+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:22.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:22.401+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:23.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:22.833+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10210 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:23.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:22.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:23.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:22.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:23.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:22.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:23.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:23.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:23.380+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:23.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:23.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:23.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:23.399+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:23.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:23.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:23.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:24.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:24.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:24.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:24.368+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:24.360+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:24.360+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:24.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:24.360+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:24.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:24.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:24.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:25.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:25.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:25.326+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:25.326+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:25.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:25.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:25.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:26.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:26.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:26.365+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:26.366+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:26.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:26.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:26.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:26.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:26.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:26.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:26.909+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:27.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:27.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:27.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:27.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:27.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:27.331+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:27.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:27.331+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:27.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:27.331+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:28.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:27.834+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10294 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:27.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:28.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:27.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:28.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:27.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:28.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:28.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:28.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:28.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:28.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:28.371+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:28.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:28.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:28.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:28.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:28.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:29.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:28.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:29.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:28.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:29.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:28.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:29.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:29.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:29.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:29.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:29.347+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:30.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:29.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:30.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:29.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:30.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:29.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:30.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:30.414+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:30.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:30.414+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:30.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:30.414+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:30.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:30.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:30.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:30.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:30.346+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:30.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:30.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:30.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:31.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:31.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:31.389+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:31.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:31.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:31.340+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:31.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:31.341+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:31.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:31.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:31.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:32.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:32.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:32.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:32.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:32.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:32.374+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:32.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:32.374+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:32.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:32.374+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:33.126 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:32.836+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10376 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:33.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:32.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:33.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:32.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:33.126 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:32.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:33.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:33.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:33.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:33.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:33.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:33.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:33.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:33.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:33.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:33.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:34.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:34.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:34.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:34.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:34.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:34.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:34.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:34.315+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:34.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:34.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:34.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:34.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:34.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:34.381+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:35.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:34.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:35.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:34.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:35.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:34.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:35.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:35.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:35.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:35.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:35.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:35.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:35.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:35.307+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:35.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:35.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:35.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:35.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:36.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:35.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:36.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:35.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:36.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:35.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:36.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:36.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:36.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:36.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:36.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:36.388+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:36.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:36.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:36.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:36.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:36.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:36.270+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:36.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:36.270+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:37.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:36.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:37.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:36.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:37.216 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:36.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:37.379+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:37.379+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:37.379+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:37.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:37.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:37.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:37.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:37.506 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:37.223+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:38.189 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:37.838+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10461 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:37.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:37.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:38.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:37.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:38.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:38.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:38.367+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:38.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:38.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:38.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:38.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:38.197+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:39.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:38.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:39.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:38.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:39.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:38.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:39.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:39.325+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:39.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:39.325+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:39.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:39.325+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:39.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:39.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:39.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:39.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:39.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:40.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:39.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:40.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:39.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:40.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:39.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:40.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:40.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:40.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:40.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:40.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:40.197+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:40.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:40.197+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:40.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:40.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:40.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:40.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:40.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:40.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:41.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:40.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:41.185 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:40.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:41.186 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:40.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:41.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:41.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:41.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:41.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:41.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:41.193+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:41.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:41.193+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:41.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:41.302+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:41.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:41.302+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:41.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:41.302+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:41.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:41.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:41.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:42.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:42.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:42.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:42.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:42.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:42.174+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:42.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:42.174+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:42.343+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:42.343+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:42.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:42.343+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:42.855 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:42.839+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10542 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:43.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:42.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:43.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:42.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:43.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:42.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:43.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:43.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:43.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:43.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:43.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:43.335+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:43.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:43.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:43.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:43.215+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:43.545 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:43.215+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:44.237 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:43.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:44.238 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:43.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:44.238 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:43.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:44.332+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:44.332+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:44.332+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:44.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:44.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:44.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:44.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:44.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:44.245+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:45.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:44.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:45.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:44.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:45.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:44.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:45.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:45.341+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:45.341+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:45.341+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:45.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:45.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:45.243+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:45.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:45.243+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:46.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:45.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:46.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:45.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:46.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:45.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:46.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:46.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:46.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:46.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:46.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:46.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:46.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:46.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:46.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:46.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:46.236+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:47.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:46.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:47.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:46.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:47.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:46.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:47.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:47.347+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:47.347+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:47.347+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:47.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:47.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:47.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:47.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:48.243 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:47.841+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10626 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:48.243 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:47.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:48.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:47.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:48.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:47.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:48.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:48.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:48.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:48.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:48.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:48.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:48.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:48.251+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:48.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:48.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:48.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:48.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:48.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:48.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:49.230 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:48.905+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:49.230 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:48.906+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:49.230 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:48.906+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:49.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:49.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:49.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:49.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:49.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:49.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:49.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:49.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:49.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:49.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:49.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:49.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:49.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:49.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:50.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:49.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:50.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:49.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:50.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:49.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:50.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:50.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:50.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:50.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:50.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:50.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:50.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:50.254+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:50.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:50.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:50.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:50.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:50.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:50.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:51.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:50.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:51.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:50.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:51.281 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:50.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:51.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:51.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:51.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:51.288+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:51.288+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:51.288+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:51.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:51.289+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:52.233 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:51.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:52.233 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:51.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:52.233 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:51.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:52.241+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:52.241+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:52.241+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:52.241+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:52.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:52.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:52.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:52.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:53.251 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:52.843+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10708 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:53.251 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:52.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:53.252 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:52.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:53.252 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:52.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:53.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:53.315+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:53.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:53.315+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:53.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:53.315+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:53.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:53.259+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:53.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:53.259+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:53.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:53.259+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:53.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:53.259+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:54.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:53.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:54.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:53.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:54.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:53.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:54.377 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-19T19:28:54.383+0000 7fd50653c700 0 monclient(hunting): authenticate timed out after 300 2023-12-19T19:28:54.377 INFO:teuthology.orchestra.run.smithi055.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-19T19:28:54.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:54.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:54.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:54.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:54.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:54.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:54.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:54.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:54.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:54.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:54.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:54.279+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:54.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:54.279+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:55.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:54.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:55.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:54.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:55.030 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:54.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T19:28:55.062 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-19T19:28:55.142 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dbeaef3cb93e4c5f80a219e12f39d428 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-19T19:28:55.145 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-12-19T19:28:55.157 INFO:tasks.kclient:Unmounting kernel clients... 2023-12-19T19:28:55.157 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:28:55.157 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:28:55.192 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:28:55.192 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:28:55.252 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T19:28:55.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:55.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:55.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:55.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:55.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:55.358+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:55.302+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:55.302+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:55.302+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:55.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:55.302+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:56.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:55.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:56.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:55.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:56.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:55.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:56.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:56.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:56.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:56.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:56.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:56.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:56.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:56.290+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:56.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:56.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:56.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:56.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:56.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:56.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:57.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:56.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:57.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:56.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:57.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:56.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:57.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:57.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:57.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:57.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:57.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:57.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:57.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:57.304+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:57.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:57.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:57.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:57.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:57.768 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:57.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:58.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:28:57.845+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10793 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:28:58.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:57.774+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:58.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:57.774+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:58.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:57.774+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:58.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:58.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:58.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:58.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:58.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:58.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:58.321+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:59.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:58.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:59.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:58.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:59.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:58.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:59.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:59.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:59.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:28:59.338+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:59.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:59.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:59.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:28:59.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:28:59.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:59.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:59.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:28:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:28:59.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:00.401 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:00.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:00.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:00.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:00.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:00.299+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:00.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:00.300+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:00.773 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:00.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:00.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:00.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:00.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:00.409+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:00.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:00.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:00.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:01.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:01.445+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:01.445+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:01.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:01.445+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:01.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:01.323+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:01.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:01.323+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:01.674 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:01.323+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:01.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:01.323+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:01.806+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:01.807+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:01.807+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:02.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:02.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:02.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:02.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:02.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:02.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:02.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:02.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:02.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:02.273+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:02.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:02.273+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:03.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:02.847+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10874 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:03.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:02.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:03.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:02.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:03.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:02.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:03.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:03.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:03.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:03.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:03.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:03.314+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:03.564 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:03.314+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:03.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:03.461+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:03.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:03.461+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:03.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:03.461+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:03.845 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:03.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:03.846 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:03.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:03.846 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:03.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:04.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:04.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:04.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:04.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:04.320+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:04.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:04.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:05.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:04.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:05.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:04.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:05.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:04.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:05.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:05.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:05.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:05.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:05.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:05.367+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:05.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:05.367+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:05.743 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:05.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:05.744 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:05.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:05.744 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:05.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:05.751+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:06.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:06.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:06.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:06.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:06.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:06.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:06.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:06.364+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:06.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:06.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:06.748 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:06.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:06.749 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:06.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:07.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:06.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:07.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:06.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:07.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:06.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:07.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:07.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:07.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:07.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:07.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:07.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:07.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:07.404+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:07.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:07.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:07.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:07.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:07.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:07.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:08.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:07.848+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 10958 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:08.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:07.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:08.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:07.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:08.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:07.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:08.388 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:08.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:08.389 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:08.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:08.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:08.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:08.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:08.396+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:08.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:08.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:08.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:08.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:08.757 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:08.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:09.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:08.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:09.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:08.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:09.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:08.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:09.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:09.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:09.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:09.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:09.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:09.351+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:09.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:09.351+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:09.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:09.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:09.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:09.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:09.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:09.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:10.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:09.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:10.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:09.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:10.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:09.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:10.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:10.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:10.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:10.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:10.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:10.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:10.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:10.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:10.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:10.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:10.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:10.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:10.769 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:10.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:11.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:10.777+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:11.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:10.777+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:11.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:10.777+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:11.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:11.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:11.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:11.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:11.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:11.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:11.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:11.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:11.316+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:11.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:11.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:11.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:11.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:11.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:11.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:12.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:12.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:12.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:12.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:12.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:12.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:12.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:12.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:12.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:12.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:12.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:12.296+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:12.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:12.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:13.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:12.850+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11040 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:13.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:12.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:13.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:12.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:13.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:12.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:13.564 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:13.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:13.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:13.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:13.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:13.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:13.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:13.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:13.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:13.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:13.565 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:13.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:13.566 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:13.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:14.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:13.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:14.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:13.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:14.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:13.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:14.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:14.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:14.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:14.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:14.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:14.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:14.293+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:14.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:14.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:14.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:15.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:14.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:15.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:14.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:15.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:14.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:15.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:15.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:15.338+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:15.338+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:15.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:15.522+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:15.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:15.522+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:15.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:15.522+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:16.133 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:15.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:16.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:15.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:16.134 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:15.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:16.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:16.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:16.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:16.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:16.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:16.297+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:16.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:16.297+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:16.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:16.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:16.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:16.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:16.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:16.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:17.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:16.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:17.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:16.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:17.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:16.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:17.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:17.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:17.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:17.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:17.298+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:17.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:17.299+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:17.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:17.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:17.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:17.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:17.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:17.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:18.285 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:17.852+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11125 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:18.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:17.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:18.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:17.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:18.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:17.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:18.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:18.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:18.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:18.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:18.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:18.293+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:18.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:18.293+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:18.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:18.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:18.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:18.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:18.544 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:18.517+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:19.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:18.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:19.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:18.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:19.081 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:18.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:19.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:19.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:19.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:19.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:19.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:19.324+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:19.494 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:19.324+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:19.501+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:19.501+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:19.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:20.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:19.847+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:20.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:19.847+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:20.142 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:19.847+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:20.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:20.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:20.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:20.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:20.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:20.318+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:20.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:20.318+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:20.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:20.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:20.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:20.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:20.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:20.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:21.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:21.491+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:21.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:21.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:21.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:21.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:21.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:21.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:21.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:21.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:21.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:21.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:21.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:21.307+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:21.848+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:21.848+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:22.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:21.848+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:22.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:22.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:22.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:22.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:22.543 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:22.487+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:22.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:22.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:22.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:22.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:22.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:22.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:22.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:22.332+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:23.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:22.854+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11206 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:23.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:22.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:23.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:22.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:23.383 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:22.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:23.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:23.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:23.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:23.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:23.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:23.372+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:23.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:23.372+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:23.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:23.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:23.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:24.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:24.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:24.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:24.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:24.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:24.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:24.364+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:24.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:24.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:24.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:25.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:25.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:25.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:25.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:25.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:25.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:25.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:25.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:25.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:25.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:25.346+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:26.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:25.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:26.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:26.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:26.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:26.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:26.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:26.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:26.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:26.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:26.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:26.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:26.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:26.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:26.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:26.859+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:27.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:27.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:27.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:27.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:27.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:27.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:27.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:28.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:27.855+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11290 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:27.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:27.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:27.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:28.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:28.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:28.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:28.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:28.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:28.389+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:28.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:28.389+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:28.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:28.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:28.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:28.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:28.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:28.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:29.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:28.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:29.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:28.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:29.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:28.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:29.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:29.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:29.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:29.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:29.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:29.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:29.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:29.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:29.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:29.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:29.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:29.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:29.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:30.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:29.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:30.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:29.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:30.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:29.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:30.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:30.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:30.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:30.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:30.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:30.377+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:30.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:30.377+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:30.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:30.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:30.798 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:30.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:30.799 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:30.554+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:30.806+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:30.806+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:30.806+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:31.675 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:31.538+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:31.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:31.538+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:31.676 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:31.538+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:31.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:31.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:31.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:31.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:31.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:31.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:31.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:31.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:31.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:31.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:32.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:31.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:32.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:32.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:32.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:32.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:32.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:32.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:32.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:32.421+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:32.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:32.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:32.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:32.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:32.775 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:32.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:33.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:32.857+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11372 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:33.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:32.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:33.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:32.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:33.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:32.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:33.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:33.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:33.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:33.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:33.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:33.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:33.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:33.441+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:33.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:33.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:33.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:33.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:33.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:33.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:34.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:33.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:34.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:33.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:34.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:33.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:34.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:34.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:34.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:34.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:34.731 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:34.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:34.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:34.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:34.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:34.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:34.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:34.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:35.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:34.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:35.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:34.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:35.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:34.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:35.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:35.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:35.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:35.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:35.774 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:35.550+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:35.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:35.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:35.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:35.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:35.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:35.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:35.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:35.464+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:36.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:35.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:36.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:35.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:36.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:35.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:36.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:36.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:36.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:36.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:36.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:36.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:36.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:36.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:36.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:36.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:36.491+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:36.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:36.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:36.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:37.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:37.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:37.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:37.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:37.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:37.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:37.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:37.536+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:37.851 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:37.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:37.852 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:37.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:37.852 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:37.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:37.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:37.783+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:37.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:37.783+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:37.852 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:37.783+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:38.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:37.859+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11457 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:38.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:38.641+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:38.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:38.641+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:38.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:38.641+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:38.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:38.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:38.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:38.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:38.498+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:39.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:38.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:39.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:38.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:39.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:38.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:39.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:39.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:39.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:39.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:39.760 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:39.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:39.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:39.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:39.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:39.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:39.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:39.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:39.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:39.508+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:40.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:39.768+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:40.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:39.768+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:40.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:39.768+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:40.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:40.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:40.536+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:40.536+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:40.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:40.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:40.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:40.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:40.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:40.776+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:40.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:40.776+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:40.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:40.776+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:41.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:41.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:41.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:41.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:41.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:41.759+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:41.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:41.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:41.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:41.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:42.853 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:42.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.853 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:42.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.854 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:42.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:42.854 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:42.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.854 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:42.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.854 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:42.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:42.854 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:42.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:42.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:42.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:42.855 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:42.568+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:43.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:42.861+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11538 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:44.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:44.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:44.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:44.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:43.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:43.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:43.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:43.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:43.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:43.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:44.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:43.573+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:45.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:44.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:45.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:44.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:45.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:44.809+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:45.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:44.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:45.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:44.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:45.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:44.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:45.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:44.603+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:45.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:44.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:45.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:44.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:45.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:44.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:45.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:46.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:45.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:46.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:45.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:45.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:45.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:45.561+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:45.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:45.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:45.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:45.613+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:46.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:46.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:46.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.811 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:46.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:46.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:46.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:46.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:46.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:46.812 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:46.565+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:47.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:46.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:47.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:46.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:47.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:46.819+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:47.785 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:47.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:47.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:47.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:47.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:47.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:47.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:47.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:47.558+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:48.043 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:47.863+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11622 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:48.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:47.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:47.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.044 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:47.794+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:48.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:48.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:48.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:48.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:48.750+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:48.750+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:48.750+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:48.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:48.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:48.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:48.531+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:49.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:49.737+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:49.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:49.737+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:49.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:49.737+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:49.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:49.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:49.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:49.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:49.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:49.560+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:49.560+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:49.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:49.560+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:50.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:50.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:50.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:50.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:50.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:50.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:50.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:50.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:50.508+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:50.508+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:50.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:50.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:50.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:50.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:51.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:51.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:51.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:51.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:51.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:51.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:51.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:51.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:51.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:51.576+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:51.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:51.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:51.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:51.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:51.469+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:52.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:52.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:52.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:52.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:52.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:52.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:52.794 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:52.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:52.795 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:52.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:52.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:52.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:52.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:52.796 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:52.442+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:53.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:52.864+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11704 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:53.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:53.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:53.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:53.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:53.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:53.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:53.682 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:53.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:53.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:53.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:53.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:53.418+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:53.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:53.418+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:54.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:53.690+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:54.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:53.690+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:54.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:53.690+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:54.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:54.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:54.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:54.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:54.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:54.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:54.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:54.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:54.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:54.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:54.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:54.416+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:55.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:54.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:55.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:54.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:55.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:54.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:55.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:55.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:55.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:55.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:55.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:55.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:55.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:55.397+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:55.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:55.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:55.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:56.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:55.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:56.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:55.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:56.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:55.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:56.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:56.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:56.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:56.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:56.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:56.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:56.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:56.389+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:57.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:56.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:57.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:56.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:57.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:56.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:57.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:56.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:56.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:56.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:57.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:57.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:57.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:57.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:57.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:57.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:57.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:57.431+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:58.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:29:57.866+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11789 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:29:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:57.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:57.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:57.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:58.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:58.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:58.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:58.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:58.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:58.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:58.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:58.425+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:29:59.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:58.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:59.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:58.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:59.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:58.899+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:29:59.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:58.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:59.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:58.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:59.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:58.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:29:59.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:59.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:59.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:59.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:29:59.432+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:59.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:59.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:00.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:29:59.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:00.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:59.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:00.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:59.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:00.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:29:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:29:59.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:00.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:00.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:00.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:00.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:00.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:00.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:00.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:00.429+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:00.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:00.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:01.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:00.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:01.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:01.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:01.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:01.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:01.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:01.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:01.445+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:01.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:01.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:01.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:01.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:01.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:02.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:01.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:02.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:02.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:02.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:02.490+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:02.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:02.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:02.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:02.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:02.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:03.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:02.868+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11870 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:02.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:02.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:02.803+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:03.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:03.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:03.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:03.491+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:03.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:03.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:03.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:03.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:03.846+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:03.846+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:04.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:03.846+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:04.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:04.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:04.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:04.468+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:04.711+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:04.711+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:04.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:04.711+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:05.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:04.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:05.285 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:04.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:05.286 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:04.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:05.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:05.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:05.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:05.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:05.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:05.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:05.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:05.465+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:05.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:05.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:06.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:05.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:06.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:05.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:05.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.044 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:05.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:06.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:06.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:06.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:06.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:06.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:06.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:06.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:06.442+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:06.905+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:06.905+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:06.905+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:07.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:07.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:07.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:07.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:07.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:07.696 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:07.427+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:07.987 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:07.870+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 11954 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:07.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:07.862+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.987 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:07.862+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:07.862+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:07.988 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:07.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:07.988 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:07.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:07.988 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:07.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:08.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:08.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:08.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:08.425+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:08.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:08.662+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:08.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:08.662+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:08.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:08.662+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:08.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:08.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:09.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:08.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:09.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:09.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:09.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:09.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:09.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:09.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:09.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:09.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:09.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:10.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:09.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:10.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:09.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:10.145 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:09.820+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:10.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:10.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:10.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:10.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:10.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:10.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:10.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:10.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:10.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:10.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:10.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:10.490+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:11.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:10.842+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:10.842+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:10.842+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:11.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:11.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:11.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:11.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:11.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:11.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:11.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:11.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:12.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:11.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:12.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:11.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:12.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:11.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:12.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:12.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:12.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:12.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:12.547+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:12.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:12.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:12.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:12.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:12.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:12.575+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:13.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:12.872+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12036 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:13.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:12.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:13.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:12.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:13.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:12.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:13.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:13.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:13.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:13.584+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:13.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:13.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:13.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:13.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:13.547+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:13.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:13.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:14.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:13.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:14.879 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:14.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:14.879 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:14.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:14.880 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:14.567+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:14.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:14.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:14.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:14.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:14.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:14.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:14.880 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:14.577+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:15.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:14.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:15.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:14.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:15.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:14.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:15.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:15.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:15.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:15.548+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:15.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:15.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:15.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:15.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:15.552+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:15.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:15.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:15.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:16.900 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:16.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:16.900 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:16.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:16.901 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:16.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:16.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:16.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:16.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:16.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:16.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:16.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:16.901 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:16.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:17.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:16.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:17.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:16.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:17.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:16.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:17.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:17.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:17.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:17.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:17.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:17.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:17.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:17.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:17.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:17.585+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:17.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:17.585+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:18.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:17.874+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12121 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:18.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:17.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:18.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:17.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:18.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:17.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:18.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:18.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:18.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:18.526+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:18.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:18.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:18.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:18.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:18.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:18.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:18.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:18.586+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:19.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:18.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:19.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:18.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:19.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:18.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:19.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:19.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:19.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:19.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:19.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:19.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:19.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:19.582+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:20.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:19.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:20.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:19.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:20.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:19.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:20.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:20.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:20.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:20.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:20.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:20.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:20.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:20.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:20.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:20.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:21.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:20.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:20.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:21.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:20.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:21.937 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:21.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:21.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:21.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:21.938 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:21.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:21.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:21.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:21.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:21.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:21.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:21.570+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:22.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:21.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:22.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:21.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:22.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:21.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:22.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:22.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:22.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:22.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:22.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:22.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:22.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:22.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:22.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:23.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:22.876+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12202 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:22.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:22.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:22.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:23.923 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:23.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:23.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:23.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:23.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:23.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:23.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:23.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:23.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:23.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:23.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:23.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:23.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:23.582+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:24.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:23.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:23.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:24.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:23.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:24.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:24.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:24.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:24.521+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:24.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:24.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:24.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:24.620+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:25.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:24.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:24.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:25.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:24.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:25.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:25.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:25.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:25.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:25.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:25.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:25.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:25.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:25.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:25.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:25.670+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:26.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:25.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:25.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:26.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:25.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:26.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:26.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:26.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:26.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:26.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:26.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:26.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:26.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:26.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:26.656+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:26.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:26.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:27.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:26.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:27.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:27.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:27.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:27.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:27.679+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:28.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:27.878+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12286 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:28.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:28.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:28.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:28.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:28.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:28.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:28.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:28.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:28.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:28.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:28.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:28.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:28.656+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:28.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:28.656+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:28.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:29.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:28.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:29.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:28.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:29.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:29.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:29.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:29.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:29.950 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:29.622+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:29.950 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:29.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:29.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:29.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:29.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:29.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:29.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:29.626+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:30.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:29.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:30.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:29.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:30.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:29.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:30.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:30.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:30.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:30.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:30.934 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:30.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:30.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:30.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:30.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:30.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:30.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:30.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:30.935 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:30.625+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:31.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:30.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:30.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:31.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:30.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:31.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:31.972 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:31.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:31.973 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:31.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:31.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:31.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:31.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:31.596+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:32.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:31.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:32.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:31.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:32.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:31.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:32.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:32.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:32.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:32.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:32.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:32.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:32.872 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:32.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:32.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:32.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:32.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:32.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:32.873 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:32.625+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:33.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:32.880+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12368 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:33.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:32.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:33.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:32.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:33.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:32.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:33.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:33.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:33.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:33.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:33.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:33.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:33.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:33.601+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:33.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:33.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:33.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:33.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:33.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:33.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:34.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:33.983+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:34.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:33.983+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:34.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:33.983+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:35.020 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:34.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:35.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:34.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:35.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:34.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:35.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:34.577+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:35.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:34.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:35.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:34.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:35.022 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:34.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:35.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:35.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:35.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:35.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:35.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:35.028+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:36.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:35.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:35.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.005 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:35.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:35.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:35.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:35.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:35.607+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:36.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:36.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:36.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:36.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:36.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:36.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:36.993 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:36.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.993 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:36.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.993 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:36.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:36.994 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:36.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.994 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:36.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.994 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:36.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:36.994 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:36.630+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:37.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:37.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:37.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:37.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:37.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:37.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:37.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:37.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:37.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:37.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:37.590+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:38.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:37.882+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12453 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:38.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:38.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:38.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:37.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:38.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:38.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:38.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:38.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:38.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:38.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:38.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:38.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:38.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:38.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:38.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:38.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:38.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:38.633+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:38.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:39.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:38.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:39.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:38.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:39.944 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:39.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:39.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:39.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:39.945 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:39.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:39.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:39.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:39.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:39.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:39.633+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:40.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:39.953+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:40.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:39.953+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:40.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:39.953+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:40.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:40.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:40.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:40.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:40.963 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:40.580+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:40.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:40.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:40.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:40.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:40.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:40.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:40.964 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:40.617+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:41.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:40.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:41.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:40.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:41.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:40.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:41.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:41.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:41.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:41.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:41.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:41.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:41.584+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:41.999 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:41.585+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:42.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:42.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:42.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:42.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:42.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:42.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:42.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:42.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:42.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:42.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:42.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:42.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:42.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:42.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:42.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:42.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:42.553+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:43.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:42.884+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12530 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:43.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:43.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:43.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:42.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:43.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:43.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:43.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:43.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:43.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:43.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:43.951 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:43.589+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:43.951 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:43.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:43.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:43.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:43.952 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:43.578+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:44.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:43.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:44.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:43.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:44.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:43.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:44.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:44.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:44.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:44.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:44.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:44.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:44.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:44.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:44.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:44.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:44.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:44.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:45.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:45.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:45.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:45.519+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:45.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:45.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:45.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:45.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:45.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:45.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:46.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:45.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:46.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:46.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:46.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:46.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:46.474+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:46.474+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:46.474+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:46.474+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:46.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:47.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:46.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:47.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:46.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:47.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:47.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:47.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:47.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:47.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:47.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:47.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:47.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:47.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:47.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:48.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:47.886+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12607 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:47.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:47.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:47.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:48.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:48.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:48.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:48.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:48.656+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:48.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:48.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:48.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:48.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:48.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:48.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:48.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:48.439+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:49.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:48.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:49.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:48.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:49.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:48.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:49.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:49.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:49.613+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:49.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:49.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:49.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:49.406+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:50.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:49.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:50.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:49.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:50.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:49.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:50.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:50.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:50.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:50.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:50.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:50.642+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:50.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:50.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:50.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:50.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:50.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:50.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:50.430+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:51.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:50.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:51.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:50.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:51.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:50.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:51.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:51.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:51.675 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:51.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:51.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:51.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:51.676 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:51.405+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:52.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:51.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:51.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:51.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:51.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:52.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:51.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:52.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:51.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:52.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:52.643+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:52.643+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:52.643+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:52.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:52.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:52.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:52.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:52.437+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:53.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:52.888+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12683 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:53.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:52.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:53.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:52.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:53.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:52.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:53.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:53.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:53.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:53.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:53.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:53.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:53.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:53.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:53.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:54.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:53.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:54.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:53.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:54.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:53.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:54.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:54.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:54.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:54.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:54.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:54.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:54.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:54.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:54.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:54.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:54.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:54.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:54.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:54.433+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:55.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:54.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:55.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:54.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:55.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:54.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-19T19:30:55.335 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2023-12-19T19:30:55.335 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:30:55.335 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:30:55.356 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-19T19:30:55.356 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:30:55.417 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T19:30:55.655 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:55.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:55.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:55.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:55.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:55.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:55.656 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:55.456+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:56.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:55.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:55.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:55.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:55.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:55.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:56.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:55.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:56.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:56.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:56.664+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:56.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:56.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:56.427+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:56.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:56.427+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:57.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:56.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:57.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:56.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:57.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:56.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:57.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:57.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:57.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:57.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:57.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:57.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:57.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:58.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:30:57.889+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12761 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:30:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:57.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:57.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:58.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:57.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:58.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:58.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:58.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:58.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:58.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:58.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:58.502+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:58.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:58.502+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:30:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:58.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:58.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:30:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:30:58.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:30:59.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:59.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:59.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:59.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:59.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:30:59.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:30:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:59.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:59.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:59.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:30:59.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:30:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:30:59.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.012+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:00.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:00.702+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:00.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:00.702+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:00.702+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:00.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:00.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:00.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:00.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:00.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:00.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:00.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:00.510+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:01.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:01.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:01.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:00.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:01.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:01.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:01.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:01.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:01.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:01.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:01.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:01.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:01.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:01.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:01.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:01.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:01.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:01.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:02.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:01.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:02.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:01.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:02.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:01.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:02.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:02.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:02.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:02.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:02.614+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:02.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:02.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:02.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:02.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:02.462+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:03.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:03.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:03.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:03.293 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:02.891+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12837 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:03.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:03.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:03.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:03.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:03.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:03.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:03.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:03.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:03.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:03.415+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:04.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:04.062+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:04.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:04.062+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:04.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:04.062+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:04.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:04.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:04.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:04.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:04.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:04.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:04.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:04.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:04.459+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:05.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:05.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:05.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:05.475 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:05.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:05.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:05.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:05.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:05.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:05.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:05.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:05.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:05.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:05.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:05.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:05.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:05.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:05.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:05.482+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:06.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:06.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:06.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:06.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:06.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:06.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:06.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:06.606+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:06.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:06.606+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:06.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:06.606+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:06.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:06.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:06.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:06.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:06.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:06.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:06.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:06.505+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:07.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:07.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:07.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:07.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:07.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:07.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:07.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:07.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:07.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:07.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:07.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:07.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:07.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:07.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:07.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:07.529+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:08.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:07.893+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12914 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:08.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:08.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:08.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:08.081+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:08.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:08.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:08.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:08.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:08.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:08.496+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:08.496+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:08.496+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:08.496+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:09.529 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:09.115+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:09.529 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:09.115+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:09.529 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:09.115+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:09.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:09.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:09.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:09.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:09.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:09.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:09.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:09.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:09.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:09.537+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:10.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:10.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:10.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:10.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:10.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:10.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:10.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:10.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:10.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:10.563+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:11.533 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:11.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:11.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:11.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:11.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:11.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:11.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:11.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:11.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:11.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:11.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:11.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:11.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:11.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:11.542+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:12.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:12.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:12.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:12.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:12.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:12.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:12.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:12.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:12.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:12.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:12.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:12.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:12.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:12.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:12.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:12.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:12.550+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:13.197 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:12.895+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 12990 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:13.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:13.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:13.509 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:13.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:13.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:13.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:13.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:13.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:13.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:13.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:13.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:13.517+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:13.517+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:14.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:14.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:14.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:14.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:14.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:14.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:14.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:14.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:14.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:14.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:14.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:14.553+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:15.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:15.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:15.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:15.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:15.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:15.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:15.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:15.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:15.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:15.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:15.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:15.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:15.600+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:16.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:16.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:16.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:16.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:16.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:17.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:16.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:16.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:16.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:16.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:16.630+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:17.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:17.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:17.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:17.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:17.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:17.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:17.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:17.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:17.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:17.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:17.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:17.653+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:18.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:17.897+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13068 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:18.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:17.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:18.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:17.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:18.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:17.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:18.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:18.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:18.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:18.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:18.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:18.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:18.663+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:19.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:18.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:19.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:18.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:19.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:18.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:19.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:19.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:19.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:19.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:19.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:19.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:19.638+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:20.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:19.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:20.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:19.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:20.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:19.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:20.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:20.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:20.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:20.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:20.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:20.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:20.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:20.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:20.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:20.604+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:20.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:20.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:20.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:21.226+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:21.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:21.226+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:21.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:21.226+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:21.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:21.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:21.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:21.600+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:21.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:21.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:21.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:22.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:22.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:22.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:22.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:22.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:22.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:22.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:22.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:22.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:22.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:22.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:22.555+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:22.892 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:22.555+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:23.196 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:22.899+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13144 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:23.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:23.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:23.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:23.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:23.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:23.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:23.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:23.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:23.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:23.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:23.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:23.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:23.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:24.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:24.193+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:24.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:24.193+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:24.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:24.193+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:24.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:24.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:24.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:24.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:24.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:24.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:24.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:24.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:24.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:24.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:24.498+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:25.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:25.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:25.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:25.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:25.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:25.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:25.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:25.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:25.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:25.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:25.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:25.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:25.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:25.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:25.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:25.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:25.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:26.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:26.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:26.497 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:26.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:26.498 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:26.138+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:26.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:26.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:26.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:26.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:26.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:26.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:26.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:26.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:26.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:26.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:26.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:27.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:27.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:27.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:27.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:27.451 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:27.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:27.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:27.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:27.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:27.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:27.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:27.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:27.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:27.459+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:28.017 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:27.901+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13221 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:28.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:27.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:28.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:27.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:28.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:27.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:28.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:28.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:28.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:28.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:28.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:28.722 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:28.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:28.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:28.410+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:28.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:28.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:28.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:29.417 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:29.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:29.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:29.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:29.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:29.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:29.751 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:29.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:29.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:29.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:29.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:29.426+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:29.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:29.426+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:29.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:29.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:29.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:30.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:30.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:30.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:30.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:30.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:30.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:30.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:30.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:30.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:30.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:30.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:30.398+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:30.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:30.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:31.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:30.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:31.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:31.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:31.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:31.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:31.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:31.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:31.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:31.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:31.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:31.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:31.544 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:31.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:31.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:31.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:31.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:32.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:32.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:32.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:32.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:32.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:32.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:32.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:32.363+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:33.163 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:32.903+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13297 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:33.163 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:32.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:33.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:32.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:33.164 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:32.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:33.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:33.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:33.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:33.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:33.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:33.344+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:33.344+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:33.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:33.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:33.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:34.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:34.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:34.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:34.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:34.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:34.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:34.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:34.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:34.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:34.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:34.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:34.315+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:35.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:34.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:35.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:34.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:35.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:34.814+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:35.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:35.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:35.349+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:35.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:35.349+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:35.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:35.349+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:35.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:35.349+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:36.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:35.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:36.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:35.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:36.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:35.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:36.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:36.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:36.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:36.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:36.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:36.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:36.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:36.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:36.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:36.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:36.382+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:36.799+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:36.799+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:36.799+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:37.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:37.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:37.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:37.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:37.372 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:37.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:37.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:37.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:37.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:37.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:37.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:37.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:37.775 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:37.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:38.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:37.905+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13375 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:38.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:37.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:38.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:37.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:38.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:37.783+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:38.148+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:38.148+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:38.412 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:38.148+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:38.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:38.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:38.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:38.421+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:39.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:38.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:39.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:38.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:39.145 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:38.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:39.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:39.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:39.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:39.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:39.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:39.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:39.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:39.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:39.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:39.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:39.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:39.397+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:39.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:39.397+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:40.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:39.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:40.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:39.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:40.122 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:39.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:40.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:40.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:40.436 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:40.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:40.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:40.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:40.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:40.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:40.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:40.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:40.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:40.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:40.446+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:41.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:40.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:41.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:40.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:41.171 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:40.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:41.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:41.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:41.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:41.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:41.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:41.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:41.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:41.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:41.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:41.451+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:42.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:41.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:42.141 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:41.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:42.142 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:41.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:42.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:42.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:42.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:42.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:42.415 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:42.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:42.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:42.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:42.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:42.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:42.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:43.097 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:42.907+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13451 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:43.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:42.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:43.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:42.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:43.098 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:42.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:43.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:43.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:43.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:43.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:43.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:43.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:43.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:43.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:43.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:43.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:43.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:44.131 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:43.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:44.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:43.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:44.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:43.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:44.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:44.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:44.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:44.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:44.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:44.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:44.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:45.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:44.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:45.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:44.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:45.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:44.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:45.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:45.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:45.505 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:45.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:45.506 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:45.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:45.514+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:46.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:45.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:46.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:45.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:46.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:45.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:46.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:46.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:46.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:46.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:46.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:46.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:46.544+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:46.544+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:46.544+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:46.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:46.545+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:47.209 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:46.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:46.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.210 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:46.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:47.218+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:47.218+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:47.218+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:47.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:47.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:47.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:47.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:47.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:47.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:47.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:47.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:47.579+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:48.193 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:47.909+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13528 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:48.545 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:48.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:48.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:48.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:48.546 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:48.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:48.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:48.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:48.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:48.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:48.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:48.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:48.569+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:49.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:49.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:49.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:49.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:49.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:49.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:50.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:49.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:50.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:49.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:50.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:49.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:49.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:49.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:49.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:49.574+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:50.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:50.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:50.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:50.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:50.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:50.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:51.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:50.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:50.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:50.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:50.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:50.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:50.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:50.612+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:51.215+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:51.215+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:51.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:51.215+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:52.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:51.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:51.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:51.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:51.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:51.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:51.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:51.609+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:52.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:52.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:52.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:52.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:52.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:52.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:52.903 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:52.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:52.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:52.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:52.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:52.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:52.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:52.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:52.601+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:53.159 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:52.911+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13604 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:53.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:53.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:53.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:53.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:53.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:53.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:54.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:53.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:53.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:53.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:53.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:53.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:53.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:53.578+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:54.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:54.213+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:54.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:54.213+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:54.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:54.213+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:55.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:54.720+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:54.720+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:54.720+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:54.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:54.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:54.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:54.623+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:55.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:55.214+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:55.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:55.214+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:55.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:55.214+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:55.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.877 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:55.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:55.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.877 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:55.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:55.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:55.878 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:55.633+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:56.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:56.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:56.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:56.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:56.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:56.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:57.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:56.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:56.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:56.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:56.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:56.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:56.612+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:56.612+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:57.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:57.219+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:57.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:57.219+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:57.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:57.219+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:57.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:57.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:57.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:57.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:57.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:57.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:57.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:57.602+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:58.253 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:31:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:31:57.914+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13682 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:31:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:58.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:58.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:58.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:58.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:58.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:58.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:31:59.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:58.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:59.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:58.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:59.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:58.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:31:59.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:58.620+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:31:59.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:59.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:59.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:59.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:31:59.543 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:31:59.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:59.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:59.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:31:59.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:59.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:00.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:59.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:00.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:59.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:00.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:31:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:31:59.620+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:00.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:00.266+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:00.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:00.266+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:00.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:00.266+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:01.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:00.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:01.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:00.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:01.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:00.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:00.665+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:00.665+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:00.665+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:00.665+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:01.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:01.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:01.279+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:01.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:01.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:01.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:01.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:01.621+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:01.670+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:01.670+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:01.958 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:01.670+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:02.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:02.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:02.298+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:02.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:02.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:02.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:02.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:02.907 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:02.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:02.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:02.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:02.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:02.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:02.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:02.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:02.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:02.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:03.269 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:02.916+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13758 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:03.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:03.277+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:03.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:03.277+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:03.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:03.277+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:03.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:03.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:03.601+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:03.601+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:04.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:03.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:04.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:03.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:04.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:03.730+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:04.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:04.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:04.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:04.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:04.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:04.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:04.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:04.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:04.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:04.569+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:05.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:04.682+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:05.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:04.682+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:05.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:04.682+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:05.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:05.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:05.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:05.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:05.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:05.289+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:05.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:05.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:05.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:05.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:06.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:05.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:06.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:05.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:06.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:05.607+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:06.564 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:06.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:06.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:06.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:06.565 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:06.320+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:06.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:06.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:06.695+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:06.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:06.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:06.573+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:06.573+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:07.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:07.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:07.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:07.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:07.608 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:07.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:07.909 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:07.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:07.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:07.654+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:07.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:07.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:07.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:07.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:07.617+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:08.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:07.918+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13831 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:08.676 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:08.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:08.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:08.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:08.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:08.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:08.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:08.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:08.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:08.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:08.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:08.592+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:08.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:08.592+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:08.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:08.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:08.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:09.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:09.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:09.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:09.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:09.614 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:09.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:09.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:09.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:09.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:09.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:09.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:09.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:09.623+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:10.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:10.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:10.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:10.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:10.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:10.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:10.623+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:11.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:10.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:11.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:10.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:11.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:10.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:11.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:11.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:11.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:11.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:11.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:11.322+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:11.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:11.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:11.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:11.610+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:12.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:11.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:11.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:11.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:12.281+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:12.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:12.281+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:12.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:12.281+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:12.911 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:12.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:12.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:12.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:12.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:12.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:12.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:12.912 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:12.638+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:13.238 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:12.920+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13907 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:13.247+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:13.247+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:13.247+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:14.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:13.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:14.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:13.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:14.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:13.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:14.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:13.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:14.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:13.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:14.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:13.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:14.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:13.668+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:14.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:14.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:14.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:14.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:14.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:14.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:14.711+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:14.711+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:14.711+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:15.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:14.711+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:15.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:15.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:16.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:15.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:16.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:15.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:16.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:15.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:16.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:15.720+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:16.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:16.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:16.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:16.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:16.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:16.245+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:16.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:16.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:16.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:16.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:16.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:16.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:16.684+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:17.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:17.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:17.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:17.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:17.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:17.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:17.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:17.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:17.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:17.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:17.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:17.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:17.701+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:18.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:17.922+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 13985 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:18.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:18.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:18.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:18.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:19.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:18.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:19.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:18.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:19.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:18.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:18.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:18.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:18.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:18.667+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:19.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:19.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:19.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:20.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:19.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:20.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:19.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:20.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:19.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:19.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:19.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:19.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:19.690+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:20.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:20.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:20.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:20.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:20.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:20.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:20.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:20.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:20.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:20.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:20.757 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:20.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:21.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:20.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:21.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:20.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:20.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:21.217+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:21.217+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:21.217+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:21.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:21.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:21.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:21.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:21.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:21.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:21.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:22.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:22.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:22.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:22.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:22.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:22.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:22.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:22.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:22.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:22.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:22.590+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:22.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:22.591+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:23.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:22.924+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14061 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:23.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:23.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:23.133+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:23.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:23.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:23.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:23.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:23.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:23.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:23.609+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:23.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:23.609+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:24.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:24.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:24.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:24.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:24.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:24.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:24.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:24.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:24.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:24.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:24.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:25.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:24.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:25.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:24.653+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:25.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:25.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:25.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:25.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:25.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:25.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:25.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:25.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:25.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:25.652+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:26.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:25.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:26.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:25.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:26.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:25.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:26.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:26.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:26.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:27.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:26.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:26.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:26.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:26.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:26.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:26.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.045 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:26.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:27.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:27.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:27.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:27.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:27.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:27.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:27.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:27.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:27.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:27.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:27.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:27.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:27.649+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:28.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:27.926+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14138 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:28.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:28.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:28.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:28.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:28.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:29.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:28.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:29.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:28.625+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:29.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:29.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:29.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:29.660+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:29.660+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:30.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:29.660+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:30.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:29.660+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:30.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:30.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:30.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:30.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:30.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:30.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:30.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:30.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:30.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:31.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:30.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:31.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:30.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:31.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:30.633+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:31.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:30.633+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:31.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:31.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:31.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:31.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:31.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:31.728+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:32.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:31.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:31.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:31.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:31.598+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:32.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:32.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:32.056+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:32.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:32.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:32.606+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:32.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:32.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:32.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:32.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:32.921 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:32.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:33.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:32.928+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14214 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:33.057+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:33.057+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:33.057+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:34.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:33.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:34.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:33.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:34.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:33.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:34.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:33.643+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:34.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:34.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:34.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:34.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:34.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:34.089+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:34.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:34.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:34.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:34.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:34.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:34.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:35.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:34.617+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:35.120+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:35.120+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:35.120+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:35.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:35.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:35.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:36.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:35.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:35.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:35.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:35.582+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:36.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:36.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:36.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:36.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:36.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:36.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:36.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:36.614+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:37.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:36.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:36.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:36.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:37.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:37.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:37.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:37.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:37.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:37.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:37.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:37.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:37.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:37.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:37.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:37.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:38.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:37.931+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14292 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:38.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:38.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:38.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:38.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:38.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:38.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:38.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:38.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:38.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:38.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:38.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:38.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:38.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:38.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:38.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:38.886 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:38.603+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:39.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:39.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:39.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:40.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:39.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:40.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:39.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:40.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:39.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:40.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:39.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:40.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:39.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:40.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:39.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:40.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:39.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:40.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:40.054+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:40.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:40.054+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:40.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:40.054+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:40.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:40.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:40.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:40.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:40.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:40.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:40.670+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:41.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:41.091+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:41.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:41.091+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:41.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:41.091+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:41.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:41.815+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:41.815+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.957 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:41.815+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:41.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:41.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:41.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:41.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:41.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:41.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:42.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:42.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:42.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:42.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:42.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:42.069+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:42.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:42.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:42.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:42.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:42.924 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:42.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:42.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:42.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:42.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:42.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:42.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:42.623+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:42.925 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:42.623+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:43.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:42.933+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14368 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:43.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:43.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:43.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:43.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:43.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:43.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:44.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:44.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:44.023 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:44.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:43.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:44.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:43.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:44.024 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:43.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:44.025 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:43.662+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:44.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:44.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:44.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:44.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:44.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:44.031+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:45.001 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:44.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:44.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:44.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:44.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:44.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:44.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.002 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:44.619+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:45.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:45.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:45.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:45.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:45.891 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:45.605+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:46.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:46.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:46.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:46.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:45.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:46.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:45.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:46.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:45.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:46.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:46.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:46.940 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:46.941 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:46.646+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:47.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:47.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:47.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:47.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:46.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:47.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:46.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:47.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:46.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:47.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:47.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:47.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:47.611+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:48.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:47.935+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14445 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:48.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:47.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:48.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:47.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:47.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:48.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:48.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:48.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:48.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:48.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:48.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:48.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:48.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:48.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:48.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:48.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:48.607+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:48.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:48.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:48.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:49.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:49.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:49.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:49.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:49.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:49.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:49.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:49.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:49.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:49.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:49.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:49.616+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:49.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:49.616+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:50.156 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:50.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:50.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:50.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:50.157 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:50.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:50.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:49.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:50.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:49.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:50.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:49.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:50.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:50.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:50.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:50.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:50.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:50.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:50.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:50.577+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:51.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:50.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:51.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:50.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:51.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:50.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:51.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:51.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:51.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:51.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:51.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:51.072+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:51.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:51.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:51.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:51.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:51.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:51.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:51.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:51.528+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:52.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:51.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:52.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:51.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:52.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:51.934+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:52.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:52.032+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:52.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:52.032+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:52.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:52.032+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:52.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:52.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:52.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:52.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:52.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:52.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:52.905 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:52.557+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:53.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:52.937+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14521 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:53.063+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:53.063+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:53.063+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:53.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:52.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:53.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:52.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:53.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:52.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:53.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:53.579+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:53.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:53.579+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:53.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:53.579+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:53.946 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:53.579+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:54.244 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:53.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:54.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:53.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:54.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:53.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:54.245 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:54.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:54.245 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:54.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:54.245 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:54.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:54.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:54.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:54.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:54.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:54.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:54.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:54.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:54.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:55.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:55.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:55.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:55.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:55.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:55.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:55.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:54.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:55.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:54.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:55.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:54.977+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-19T19:32:55.500 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-12-19T19:32:55.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:32:55.500 DEBUG:teuthology.orchestra.run.smithi055:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-12-19T19:32:55.536 INFO:tasks.workunit.client.0.smithi055.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/mnt.0/client.0’: Input/output error 2023-12-19T19:32:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T19:32:55.538 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2023-12-19T19:32:55.538 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-19T19:32:55.539 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi055.front.sepia.ceph.com 2023-12-19T19:32:55.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-19T19:32:55.539 DEBUG:teuthology.orchestra.run.smithi055:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-19T19:32:55.602 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-12-19T19:32:55.602 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-19T19:32:55.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:55.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:55.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:55.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:55.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:55.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:55.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:55.613+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:56.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:56.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:56.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:55.956+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:55.956+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:56.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:55.956+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:56.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:56.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:56.920 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:56.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:56.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:56.604+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:56.921 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:56.604+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:57.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:57.037+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:57.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:57.037+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:57.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:57.037+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:57.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:56.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:57.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:56.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:57.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:56.929+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:57.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:57.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:57.617+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:57.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:57.617+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:58.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:32:57.939+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14599 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:32:58.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:58.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:58.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:58.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:58.349 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:58.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:58.349 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:58.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:58.350 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:57.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:58.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:58.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:58.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:58.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:58.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:58.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:58.922 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:58.582+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:32:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:32:59.047+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:32:59.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:58.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:59.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:58.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:59.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:58.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:32:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:59.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:59.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:59.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:32:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:32:59.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:00.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:00.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:00.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:59.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:00.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:59.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:00.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:32:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:32:59.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:00.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:00.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:00.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:00.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:00.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:00.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:00.518+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:01.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:01.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:01.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:01.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:01.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:01.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:01.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:00.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:01.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:00.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:01.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:00.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:01.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:01.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:01.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:01.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:01.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:01.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:01.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:01.505+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:02.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:02.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:02.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:02.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:02.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:02.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:02.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:01.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:02.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:01.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:02.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:01.954+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:02.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:02.486+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:02.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:02.486+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:02.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:02.486+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:02.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:02.486+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:03.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:02.942+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14675 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:03.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:02.908+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:03.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:02.908+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:03.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:02.908+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:03.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:03.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:03.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:03.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:03.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:03.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:03.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:03.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:03.441+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:03.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:03.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:03.885+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:04.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:04.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:04.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:04.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:04.457+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:04.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:04.457+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:04.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:04.457+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:04.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:04.457+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:05.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:04.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:05.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:04.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:05.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:04.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:05.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:05.114+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:05.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:05.114+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:05.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:05.114+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:05.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:05.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:05.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:05.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:05.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:05.458+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:05.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:05.458+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:06.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:05.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:06.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:05.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:06.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:05.832+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:06.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:06.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:06.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:06.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:06.406 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:06.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:06.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:06.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:06.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:06.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:06.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:06.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:06.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:06.415+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:07.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:06.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:07.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:06.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:07.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:06.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:07.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:07.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:07.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:07.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:07.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:07.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:07.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:07.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:07.380+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:08.103 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:07.944+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14752 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:08.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:07.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:08.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:07.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:08.103 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:07.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:08.359 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:08.112+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:08.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:08.112+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:08.360 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:08.112+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:08.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:08.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:08.369+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:08.369+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:09.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:08.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:09.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:08.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:09.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:08.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:09.390 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:09.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:09.391 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:09.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:09.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:09.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:09.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:09.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:09.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:09.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:09.400+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:10.111 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:09.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:10.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:09.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:10.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:09.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:10.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:10.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:10.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:10.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:10.430 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:10.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:10.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:10.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:10.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:10.440+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:11.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:10.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:11.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:10.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:11.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:10.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:11.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:11.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:11.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:11.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:11.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:11.080+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:11.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:11.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:11.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:11.473+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:11.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:11.473+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:11.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:11.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:11.809+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.045+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.045+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:12.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.045+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:12.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:12.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:12.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:12.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:12.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:12.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:12.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:12.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:13.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:12.947+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14828 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:13.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:13.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:13.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:13.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:12.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:13.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:12.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:13.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:12.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:13.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:13.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:13.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:13.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:13.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:13.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:13.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:13.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:13.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:13.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:13.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:13.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:13.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:14.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:13.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:14.763 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:14.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:14.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:14.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:14.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:14.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:14.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:14.448+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:14.772+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:14.772+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:14.772+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:15.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:15.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:15.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:15.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:15.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:15.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:15.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:15.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:15.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:15.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:15.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:15.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:15.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:15.463+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:15.996 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:15.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:15.996 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:15.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:15.996 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:15.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:16.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:16.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:16.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:16.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:16.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:16.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:16.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:16.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:16.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:16.765 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:16.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:16.766 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:16.435+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:17.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:16.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:17.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:16.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:17.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:16.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:17.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:17.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:17.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:17.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:17.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:17.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:17.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:17.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:17.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:17.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:17.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:17.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:17.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:17.394+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:18.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:17.949+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14906 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:18.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:17.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:18.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:17.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:18.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:17.794+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:18.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:18.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:18.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:18.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:18.339 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:18.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:18.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:18.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:18.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:18.348+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:19.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:18.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:19.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:18.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:19.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:18.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:19.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:19.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:19.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:19.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:19.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:19.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:19.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:19.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:19.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:19.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:19.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:20.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:19.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:20.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:19.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:20.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:19.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:20.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:20.027+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:20.027+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:20.160 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:20.027+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:20.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:20.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:20.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:20.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:20.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:21.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:21.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:21.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:21.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:21.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:21.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:21.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:20.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:21.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:20.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:21.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:20.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:21.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:21.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:21.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:21.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:21.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:21.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:21.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:21.406+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:21.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:21.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:21.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:21.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:21.998 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:21.911+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:22.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:22.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:22.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:22.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:22.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:22.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:22.406+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:23.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:22.951+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 14982 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.020+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.020+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.020+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:23.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:22.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:23.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:22.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:23.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:22.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:23.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:23.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:23.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:23.376+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:24.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:24.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:24.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:23.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:24.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:23.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:24.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:23.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:24.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:23.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:24.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:24.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:24.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:24.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:25.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:25.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:25.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:25.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:25.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:25.030+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:25.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:24.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:25.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:24.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:25.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:24.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:25.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:25.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:25.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:25.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:25.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:25.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:25.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:25.380+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:26.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:26.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:26.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:26.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:26.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:26.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:25.933+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:26.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:26.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:26.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:26.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:26.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:26.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:26.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:26.416+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:27.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:26.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:27.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:26.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:27.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:26.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:27.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:26.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:27.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:27.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:27.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:27.463+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:28.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:27.954+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15059 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:27.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:27.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:27.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:28.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:27.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:28.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:27.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:28.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:27.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:28.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:28.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:28.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:28.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:28.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:28.439+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:28.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:28.439+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:28.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:28.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:28.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:28.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:28.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:28.924+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:29.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:29.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:29.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:29.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:29.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:29.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:29.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:29.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:30.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:29.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:30.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:29.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:30.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:29.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:30.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:29.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:30.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:29.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:30.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:29.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:30.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:30.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:30.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:30.435+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:31.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:30.947+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:31.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:30.947+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:31.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:30.947+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:31.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:30.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:31.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:30.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:31.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:30.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:31.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:31.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:31.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:31.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:31.692 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:31.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:31.693 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:31.413+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:31.980 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:31.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:31.980 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:31.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:31.980 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:31.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:32.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:31.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:32.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:31.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:32.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:31.989+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:32.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:32.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:32.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:32.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:32.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:32.463+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:32.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:32.463+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:33.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:32.956+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15135 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:32.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:32.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:33.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:32.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:33.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:32.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:33.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:32.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:33.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:32.965+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:33.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:33.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:33.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:33.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:33.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:33.425+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:33.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:33.425+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:34.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:33.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:34.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:33.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:34.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:33.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:34.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:33.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:34.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:33.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:34.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:33.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:34.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:34.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:34.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:34.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:34.404+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:35.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:34.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:35.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:34.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:35.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:34.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:35.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:34.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:35.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:34.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:35.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:34.988+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:35.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:35.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:35.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:35.383+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:35.383+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:36.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:35.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:36.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:35.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:36.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:35.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:36.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:35.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:36.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:35.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:36.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:35.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:36.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:36.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:36.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:36.415+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:36.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:36.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:36.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:36.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:36.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:37.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:36.960+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:37.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:37.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:37.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:37.408+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:38.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:37.958+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15213 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:37.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:37.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:38.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:37.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:38.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:37.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:38.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:37.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:38.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:37.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:38.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:38.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:38.431+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:38.431+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:38.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:38.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:38.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:38.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:38.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:39.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:38.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:39.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:39.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:39.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:39.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:39.456+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:39.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:39.456+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:40.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:39.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:40.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:39.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:40.184 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:39.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:40.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:39.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:40.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:39.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:40.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:39.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:40.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:40.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:40.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:40.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:40.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:40.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:40.729 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:40.484+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:40.986 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:40.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:40.986 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:40.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:40.986 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:40.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:41.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:40.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:41.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:40.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:41.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:40.995+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:41.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:41.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:41.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:41.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:41.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:41.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:42.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:41.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:42.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:41.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:42.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:41.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:42.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:42.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:42.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:42.007+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:42.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:42.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:42.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:42.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:42.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:42.557+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:42.837 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:42.558+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:43.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:42.961+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15289 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:43.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:42.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:43.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:42.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:43.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:42.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:43.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:43.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:43.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:43.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:43.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:43.017+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:43.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:43.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:43.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:43.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:43.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:43.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:43.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:44.294 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:44.295 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:43.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:44.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:44.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:44.295 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:44.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:44.296 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:44.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:44.807 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:44.566+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:44.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:44.566+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:44.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:44.566+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:44.808 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:44.566+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:45.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:44.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:45.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:44.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:45.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:44.816+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:45.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:45.010+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:45.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:45.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:46.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:45.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:45.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:45.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:45.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:45.596+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:46.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:46.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:46.048+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:46.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:46.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:46.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:46.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:46.819 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:46.552+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:47.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:47.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:47.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:47.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:47.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:46.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:47.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:46.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:47.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:46.827+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:47.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:47.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:47.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:47.556+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:47.556+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:48.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:47.963+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15365 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:48.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:48.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:48.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:48.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:48.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:48.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:47.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:47.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:48.293 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:47.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:48.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:48.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:48.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:48.532+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:49.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:49.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:49.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:49.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:49.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:49.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:48.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:48.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:48.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:49.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:49.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:49.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:49.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:49.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:49.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:49.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:49.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:49.523+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:50.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:50.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:50.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:50.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:50.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:50.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:50.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:50.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:50.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:50.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:50.534+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:50.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:50.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:50.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:50.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:50.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:50.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:51.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:51.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:51.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:51.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:51.368 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:51.066+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:51.641 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:51.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:51.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:51.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:51.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:51.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:51.642 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:51.568+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:51.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:52.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:51.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:52.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:51.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:52.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:52.096+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:52.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:52.096+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:52.523 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:52.096+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:52.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:52.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:52.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:52.531+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:52.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:52.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:52.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:52.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:53.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:52.965+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15441 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:53.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:53.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:53.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:53.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:53.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:53.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:53.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:53.570+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:54.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:53.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:54.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:53.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:54.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:53.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:54.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:54.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:54.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:54.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:54.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:54.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:55.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:54.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:55.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:54.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:54.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:54.578+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:55.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:54.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:55.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:54.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:55.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:54.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:55.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:55.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:55.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:55.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:55.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:55.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:55.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:55.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:55.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:55.625+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:55.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:56.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:55.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:56.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:55.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:56.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:56.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:56.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:56.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:56.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:56.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:57.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:56.669+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:56.669+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:56.669+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.045 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:56.669+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:56.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:56.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:57.046 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:56.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:57.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:57.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:57.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:57.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:57.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:57.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:57.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:58.042 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:33:57.967+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15518 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:33:58.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:57.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:58.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:57.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:58.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:57.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:58.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:58.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:58.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:58.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:58.724+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:58.724+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:58.724+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:58.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:58.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:58.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:33:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:58.631+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:33:59.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:59.147+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:59.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:59.147+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:33:59.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:33:59.147+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:00.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:59.650+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:59.650+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:59.650+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:33:59.650+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:59.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:59.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:33:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:33:59.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:00.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:00.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:00.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:00.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:00.118+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:00.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:00.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:00.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:00.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:01.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:00.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:01.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:00.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:01.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:00.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:01.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:01.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:01.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:01.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:01.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:01.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:01.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:01.662+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:01.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:01.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:01.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:02.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:02.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:02.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:02.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:02.960 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:02.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:02.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.961 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:02.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:02.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:02.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.961 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:02.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:02.962 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:02.709+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:03.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:02.969+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15594 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:03.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:03.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:03.116+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:03.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:03.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:03.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:03.703+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:03.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:04.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:03.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:04.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:03.760+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:04.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:04.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:04.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:04.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:04.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:04.085+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:04.677+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:04.677+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:04.677+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:04.677+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:04.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:05.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:04.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:05.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:04.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:05.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:05.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:05.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:05.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:05.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:05.065+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:06.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:05.659+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:05.659+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:05.659+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:05.659+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:05.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:05.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:05.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:06.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:06.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:06.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:06.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:06.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:06.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:06.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:06.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:06.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:06.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:06.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:06.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:07.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:06.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:07.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:07.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:07.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:07.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:07.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:07.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:07.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:07.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:07.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:07.610+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:08.087 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:07.972+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15670 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:08.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:07.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:08.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:07.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:08.088 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:07.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:08.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:08.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:08.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:08.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:08.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:08.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:09.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:08.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:08.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:08.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:09.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:08.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:09.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:08.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:09.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:08.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:09.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:08.641+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:09.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:09.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:09.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:09.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:09.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:09.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:09.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:09.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:09.806+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:10.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:09.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:10.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:09.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:09.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:09.684+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:10.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:10.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:10.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:10.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:10.191+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:11.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:10.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:10.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:10.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:10.679+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:10.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:10.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:10.822+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:11.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:11.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:11.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:11.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:11.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:11.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:11.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:12.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:11.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:12.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:11.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:11.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:11.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:11.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:12.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:11.729+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:12.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:12.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:12.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:12.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:12.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:12.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:13.041 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:12.974+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15746 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:13.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:12.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:13.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:12.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:13.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:12.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:13.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:12.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:13.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:12.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:13.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:12.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:13.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:12.752+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:13.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:13.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:14.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:13.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:14.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:13.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:14.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:13.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:14.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:14.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:14.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:14.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:13.764+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:14.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:14.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:14.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:14.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:14.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:14.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:14.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:14.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:14.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:14.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:14.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:14.729+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:15.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:15.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:15.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:15.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:15.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:15.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:15.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:15.887+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:15.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:15.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:15.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:16.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:15.754+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:16.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:16.297+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:16.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:16.297+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:16.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:16.297+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:17.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:16.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:16.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:16.837+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:16.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:16.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:16.753+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:16.754+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:17.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:17.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:17.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:17.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:17.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:17.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:17.980 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:17.977+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15823 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:17.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:17.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:17.792+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:17.981 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:17.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.981 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:17.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.981 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:17.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:17.981 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:17.732+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:18.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:18.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:18.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:18.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:18.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:18.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:19.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:18.793+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:18.793+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:18.793+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:18.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:18.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:18.732+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:18.732+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:19.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:19.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:19.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:19.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:19.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:19.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:19.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:19.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:19.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:19.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:19.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:19.701+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:20.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:20.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:20.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:20.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:20.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:20.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:21.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:20.717+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:21.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:20.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:20.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:21.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:20.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:21.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:21.250+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:21.251+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:21.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:21.251+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:21.671+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:21.671+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:21.671+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:21.671+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:21.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:21.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:21.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:22.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:22.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:22.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:22.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:22.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:22.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.970 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:22.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:22.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:22.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:22.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:22.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:22.702+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:23.251 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:22.979+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15898 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:23.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:23.260+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:23.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:23.260+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:23.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:23.260+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:23.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:23.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:24.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:23.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:24.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:23.719+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:24.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:23.766+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:24.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:23.766+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:24.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:23.766+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:24.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:24.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:24.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:24.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:24.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:24.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:24.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:24.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:24.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:24.679+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:24.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:24.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:25.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:24.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:25.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:25.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:25.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:25.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:25.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:25.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:26.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:25.706+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:25.706+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:25.706+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:25.707+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:25.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:25.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:26.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:25.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:26.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:26.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:26.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:26.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:27.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:26.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:26.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:26.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:26.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:27.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:26.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:26.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:26.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:27.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:27.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:27.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:27.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:27.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:27.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:27.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:27.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:27.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:27.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:27.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:27.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:27.987 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:27.981+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 15975 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:28.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:28.225+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:28.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:28.225+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:28.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:28.225+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:29.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:28.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:28.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:28.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:28.684+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:29.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:28.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:29.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:28.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:29.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:28.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:29.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:29.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:29.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:29.236+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:30.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:29.676+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:29.676+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:29.676+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:29.676+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:29.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:29.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:30.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:29.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:30.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:30.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:30.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:30.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:30.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:30.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:31.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:30.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:30.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:30.662+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:30.662+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:30.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:30.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:31.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:30.737+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:31.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:31.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:31.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:31.229+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:31.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:31.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:31.653+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:31.653+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:31.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:31.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:31.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:32.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:32.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:32.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:32.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:32.280+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:32.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:32.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:32.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:32.975 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.976 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.976 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:32.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:32.976 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:32.606+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:33.267 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:32.984+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16050 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:33.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:33.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:33.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:33.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:33.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:33.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:33.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:33.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:33.587+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:34.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:34.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:33.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:34.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:34.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:34.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:34.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:34.534 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:34.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:34.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:34.543+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:34.543+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:34.543+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:34.543+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:34.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:34.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:34.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:34.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:03.362202+0000 (oldest deadline 2023-12-19T19:24:23.362202+0000) 2023-12-19T19:34:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:35.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:35.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:35.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:35.324+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:04.195982+0000 (oldest deadline 2023-12-19T19:24:24.195982+0000) 2023-12-19T19:34:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:35.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:35.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:35.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:35.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:36.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:36.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:36.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:36.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:36.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:36.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:37.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:37.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:37.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:37.530+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:38.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:37.986+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16128 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:38.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:38.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:38.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:38.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:39.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:39.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:39.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:39.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:39.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:40.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:41.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:40.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:41.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:40.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:24:08.414152+0000 (oldest deadline 2023-12-19T19:24:28.414152+0000) 2023-12-19T19:34:41.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:40.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:41.588+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:42.979 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:42.596+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:43.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:42.988+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16203 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:44.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:43.645+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:45.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:44.632+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:46.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:45.647+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:47.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:46.643+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:47.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:47.597+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:48.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:47.991+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16280 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:48.550+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:49.534+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:50.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:50.491+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:51.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:51.500+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:52.508+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:53.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:52.993+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16355 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:53.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:54.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:54.532+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:55.563+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:56.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:56.521+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:56.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:56.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:56.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:57.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:57.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:57.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:57.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:57.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:57.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:58.292 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:34:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:34:57.996+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16433 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:34:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:58.533+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:58.674+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:58.674+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:58.674+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:58.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:58.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:58.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:58.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:34:59.518+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:34:59.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:59.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:59.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:34:59.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:34:59.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:59.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:59.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:34:59.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:34:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:34:59.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:00.530+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:00.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:00.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:00.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:00.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:00.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:00.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:00.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:00.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:00.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:00.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:00.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:00.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:01.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:01.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:01.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:01.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:01.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:01.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:01.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:01.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:01.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:01.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:01.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:01.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:01.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:01.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:02.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:02.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:02.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:02.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:02.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:02.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:02.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:02.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:02.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:02.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:02.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:02.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:02.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:02.519+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:02.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:02.519+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:03.268 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:02.998+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16508 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:03.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:03.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:03.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:03.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:03.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:03.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:03.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:03.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:03.539+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:03.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:03.539+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:03.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:03.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:03.794 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:03.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:04.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:04.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:04.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:04.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:04.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:04.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:04.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:04.516+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:04.774 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:04.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:04.774 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:04.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:04.774 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:04.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:04.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:04.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:04.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:05.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:05.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:05.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:05.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:05.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:05.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:05.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:05.479+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:05.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:05.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:05.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:05.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:05.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:05.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:06.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:05.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:06.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:05.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:05.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:06.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:06.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:06.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:06.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:06.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:06.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:06.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:06.436+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:06.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:06.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:06.732 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:06.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:06.732 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:06.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:07.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:06.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:06.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:06.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:07.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:07.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:07.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:07.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:07.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:07.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:07.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:07.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:07.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:07.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:07.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:07.479+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:07.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:07.479+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:08.126 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:08.001+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16585 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:08.719+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:08.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:08.719+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:08.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:08.719+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:08.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:08.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:08.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:08.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:08.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:08.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:08.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:08.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:08.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:08.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:09.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:09.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:09.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:09.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:09.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:09.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:09.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:09.505+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:10.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:09.757+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:10.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:09.757+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:10.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:09.757+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:09.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:09.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:09.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:10.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:10.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:10.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:10.495+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:10.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:10.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:10.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:10.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:10.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:10.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:10.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:10.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:11.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:11.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:11.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:11.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:11.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:11.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:11.698 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:11.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:11.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:11.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:11.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:11.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:11.699 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:11.448+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:11.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:11.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:11.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:12.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:12.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:12.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:12.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:12.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:12.661+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:12.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:12.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:12.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:12.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:12.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:12.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:12.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:12.495+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:13.018 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:13.003+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16660 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:13.018 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:12.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.018 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:12.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.019 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:12.767+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:13.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:13.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:13.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:13.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:13.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:13.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:13.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:13.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:13.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:13.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:13.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:13.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:13.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:13.540+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:14.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:14.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:14.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:14.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:14.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:14.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:14.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:14.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:14.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:14.733+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:14.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:14.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:14.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:14.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:14.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:14.495+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:14.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:14.495+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:15.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:15.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:15.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:15.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:15.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:15.622+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:15.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:15.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:15.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:15.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:15.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:15.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:15.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:15.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:15.524+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:15.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:15.524+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:16.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:16.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:16.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:16.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:16.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:16.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:16.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:16.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:16.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:16.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:16.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:16.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:16.545+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:16.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:16.545+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:16.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:16.545+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:16.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:16.545+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:17.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:17.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:17.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:17.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:17.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:17.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:17.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:17.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:17.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:17.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:18.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:18.006+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16738 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:18.687+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:18.687+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:18.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:18.687+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:18.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:18.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:18.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:18.531+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:18.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:18.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:18.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:18.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:19.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:19.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:19.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:19.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:19.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:19.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:19.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:19.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:19.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:19.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:19.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:20.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:20.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:20.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:20.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:20.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:20.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:20.518+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:20.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:20.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:20.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:20.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:20.756+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:21.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:21.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:21.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:21.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:21.773 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:21.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:21.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:21.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:21.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:21.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:21.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:21.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:21.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:21.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:21.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:21.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:21.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:22.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:22.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:22.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:22.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:22.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:22.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:22.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:22.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:22.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:22.489+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:23.125 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:23.008+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16813 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:23.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:22.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:23.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:22.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:23.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:22.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:23.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:23.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:23.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:23.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:23.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:23.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:23.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:23.520+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:24.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:23.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:24.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:23.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:24.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:23.842+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:24.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:24.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:24.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:24.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:24.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:24.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:24.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:24.547+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:24.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:24.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:24.871+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:25.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:25.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:25.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:25.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:25.884 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:25.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:25.884 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:25.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:25.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:25.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:25.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:25.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:25.885 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:25.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:26.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:25.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:26.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:25.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:26.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:25.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:26.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:26.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:26.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:26.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:26.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:26.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:26.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:26.538+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:26.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:26.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:26.875+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:26.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:26.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:26.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:27.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:27.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:27.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:27.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:27.567+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:27.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:27.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:27.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:27.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:27.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:27.754+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:28.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:28.011+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16891 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:28.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:27.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:28.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:27.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:28.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:27.918+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:28.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:28.722+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:28.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:28.722+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:28.943 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:28.722+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:28.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:28.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:28.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:28.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:28.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:28.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:28.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:28.613+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:29.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:28.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:29.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:28.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:28.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:29.962 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:29.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:29.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:29.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:29.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:29.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:29.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:29.963 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:29.607+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:30.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:29.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:30.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:29.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:30.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:29.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:30.916 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:30.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:30.916 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:30.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:30.916 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:30.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:30.916 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:30.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:30.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:30.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:30.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:30.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:30.917 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:30.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:31.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:30.984+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:31.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:30.984+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:31.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:30.984+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:31.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:31.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:31.972 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:31.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:31.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:31.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:31.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:31.600+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:31.973 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:31.600+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:32.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:31.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:32.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:31.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:32.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:31.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:32.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:32.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:32.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:32.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:32.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:32.613+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:32.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:32.613+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:32.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:32.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:32.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:32.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:32.999 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:32.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:33.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:33.013+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 16967 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:33.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:33.007+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:33.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:33.007+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:33.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:33.007+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:33.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:33.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:33.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:33.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:33.988 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:33.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:33.988 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:33.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:33.989 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:33.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:33.989 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:33.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:33.989 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:33.628+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:34.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:34.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:34.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:34.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:34.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:34.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:35.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:34.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.014 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:34.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.015 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:34.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:34.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:34.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:34.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:34.620+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:35.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:35.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:35.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.024+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:35.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:35.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:35.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.968 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:35.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:35.968 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:35.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:35.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:35.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:35.969 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:35.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.978+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.978+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:35.978+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:36.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:36.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:36.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:36.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:36.957 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:36.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:36.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:36.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:36.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:36.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:36.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:36.624+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:36.958 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:36.624+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:37.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:36.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:37.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:36.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:37.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:36.967+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:37.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:37.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:37.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:37.655+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:37.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:37.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:37.638+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:38.132 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:38.016+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17046 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:38.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:37.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:38.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:37.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:38.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:37.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:39.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:38.652+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:39.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:38.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:38.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:38.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:38.895+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:38.895+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:38.895+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:39.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:39.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:39.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.929 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:39.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:39.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:39.686+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:39.686+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:39.686+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:39.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:39.686+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:40.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:39.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:40.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:39.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:40.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:39.938+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:41.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:40.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:41.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:40.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:41.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:40.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:40.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:40.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:40.897+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:40.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:40.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:40.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:40.731+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:41.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:41.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:41.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:41.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:41.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:41.647+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:41.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:42.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:41.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:41.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:41.735+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:43.010 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:42.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:43.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:42.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:43.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:42.826+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:43.011 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:42.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:43.011 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:42.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:43.011 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:42.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:43.012 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:42.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:43.012 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:42.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:43.012 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:42.731+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:43.012 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:42.731+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:43.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:43.019+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17122 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:44.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:43.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:44.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:43.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:44.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:43.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:43.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:43.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:43.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:43.742+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:43.742+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:44.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:43.742+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:44.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:43.742+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:44.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:44.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:44.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:45.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:44.819+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:45.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:44.819+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:45.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:44.819+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:45.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:44.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:45.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:44.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:45.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:44.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:45.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:44.756+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:46.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:45.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:46.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:45.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:46.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:45.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:46.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:45.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:46.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:45.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:46.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:45.641+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:46.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:45.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:46.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:45.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:46.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:45.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:46.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:45.760+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:47.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:46.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:47.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:46.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:47.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:46.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:47.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:46.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:46.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:46.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:46.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:46.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:46.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:46.746+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:47.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:47.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:47.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:47.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:47.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:47.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:47.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:47.747+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:47.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:47.747+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:48.012 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:47.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:48.012 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:47.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:48.013 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:47.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:48.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:48.022+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17200 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:49.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:48.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:49.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:48.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:48.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:49.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:49.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:49.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:49.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:48.757+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:48.757+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:48.757+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:49.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:48.757+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:50.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:49.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:50.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:49.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:50.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:49.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:50.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:50.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:50.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:49.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:49.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:49.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:50.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:49.768+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:51.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:50.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:51.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:50.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:51.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:50.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:51.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:50.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:51.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:50.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:51.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:50.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:50.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:50.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:51.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:50.804+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:51.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:50.804+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:51.705+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:51.705+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:51.705+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:52.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:51.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:52.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:51.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:52.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:51.720+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:52.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:51.817+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:52.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:51.817+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:52.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:51.817+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:52.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:51.817+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:52.697 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:52.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:52.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:52.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:52.698 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:52.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:53.015 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:52.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:53.015 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:52.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:53.015 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:52.707+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:53.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:52.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:53.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:52.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:53.016 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:52.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:53.016 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:52.771+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:53.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:53.024+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17276 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:54.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:53.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:54.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:53.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:54.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:53.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:54.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:53.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:54.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:53.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:54.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:53.699+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:53.813+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:54.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:53.813+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:53.813+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:54.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:53.813+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:54.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:54.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:55.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:54.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:55.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:54.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:55.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:54.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:55.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:54.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:55.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:54.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:54.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:54.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:55.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:54.789+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:55.638+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:55.638+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:56.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:55.638+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:56.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:55.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:56.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:55.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:56.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:55.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:56.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:55.797+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:56.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:55.797+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:56.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:55.797+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:56.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:55.798+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:56.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:56.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:56.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:56.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:57.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:56.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:57.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:56.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:56.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:56.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:56.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:57.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:56.795+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:57.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:57.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:57.596+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:57.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:57.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:57.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:57.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:57.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:57.639+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:58.145 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:35:58.027+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17355 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:35:58.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:57.807+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:58.145 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:57.807+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:58.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:57.807+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:58.146 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:57.807+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:59.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:58.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:59.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:58.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:59.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:58.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:35:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:58.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:58.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:58.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:35:59.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:58.840+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:35:59.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:58.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:59.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:58.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:35:59.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:58.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:59.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:59.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:35:59.644+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:59.637+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:59.637+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:00.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:35:59.637+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:59.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:00.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:59.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:00.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:59.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:00.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:35:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:35:59.840+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:01.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:00.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:01.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:00.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:01.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:00.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:01.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:00.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:01.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:00.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:01.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:00.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:00.794+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:00.794+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:00.794+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:01.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:00.794+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:01.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:01.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:01.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:01.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:01.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:02.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:01.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:01.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:01.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:02.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:01.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:02.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:01.791+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:03.020 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:02.700+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:03.020 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:02.700+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:03.020 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:02.700+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:02.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:02.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:02.659+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:02.776+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:02.776+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:03.021 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:02.776+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:03.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:02.776+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:03.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:03.029+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17431 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:03.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:03.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:03.678+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:03.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:03.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:03.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:03.788+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:03.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:04.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:03.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:04.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:03.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:05.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:04.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:05.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:04.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:05.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:04.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:04.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:04.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:04.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:04.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:04.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:05.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:04.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:05.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:04.793+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:05.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:05.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:05.711+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:05.768+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:05.768+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:05.768+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:05.768+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:05.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:06.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:05.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:06.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:05.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:06.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:06.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:06.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:06.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:06.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:06.658+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:06.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:06.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:06.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:06.727+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:07.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:07.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:07.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:07.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:07.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:07.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:07.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:07.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:07.722+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:07.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:07.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:07.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:07.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:07.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:08.047 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:08.032+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17509 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:08.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:08.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:09.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:08.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:08.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:08.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:09.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:08.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:09.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:08.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:09.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:08.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:09.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:08.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:09.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:08.765+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:10.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:09.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:10.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:09.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:10.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:09.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:10.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:09.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:10.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:09.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:10.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:09.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:10.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:09.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:09.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:09.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:10.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:09.796+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:10.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:10.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:10.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:10.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:10.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:10.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:10.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:10.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:11.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:10.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:11.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:10.764+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:11.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:11.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:11.661+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:12.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:11.677+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:12.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:11.677+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:12.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:11.677+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:11.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:12.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:11.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:12.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:11.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:12.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:11.717+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:13.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:12.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:13.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:12.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:13.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:12.679+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:12.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:12.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:12.663+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:12.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:12.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:12.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.027 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:12.735+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:13.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:13.034+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17585 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:13.923 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:13.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:13.924 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:13.764+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:13.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:13.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:13.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:13.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:13.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:13.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:13.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:13.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:13.925 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:13.691+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:14.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:14.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:14.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:14.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:14.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:14.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:14.803+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:14.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:15.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:14.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:15.043 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:14.606+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:15.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:15.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:15.636+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:15.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:15.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:15.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:15.833+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:15.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:16.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:15.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:16.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:15.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:16.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:16.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:16.679+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:16.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:16.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:16.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:16.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:16.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:16.803+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:16.803+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:17.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:17.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:17.751+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:17.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:17.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:17.701+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:17.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:17.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:17.756+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:17.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:17.756+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:18.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:18.037+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17664 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:19.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:18.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:19.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:18.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:19.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:18.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:18.749+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:18.750+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:18.750+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:18.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:18.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:19.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:18.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:19.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:18.802+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:19.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:19.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:19.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:19.736+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:19.736+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:19.736+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:19.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:19.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:19.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:20.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:19.843+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:20.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:20.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:20.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:20.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:20.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:20.635+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:20.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:20.635+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:20.879 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:20.635+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:20.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:20.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:20.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:20.887+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:21.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:21.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:22.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:21.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:22.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:21.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:22.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:21.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:22.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:21.805+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:21.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:21.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:21.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:22.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:21.874+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:23.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:22.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:23.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:22.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:23.030 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:22.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:23.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:22.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:23.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:22.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:23.031 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:22.800+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:23.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:22.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:23.031 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:22.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:23.032 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:22.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:23.032 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:22.893+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:23.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:23.039+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17740 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:24.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:23.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:24.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:23.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:24.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:23.701+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:24.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:23.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:24.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:23.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:24.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:23.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:24.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:23.915+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:24.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:23.915+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:24.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:23.915+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:24.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:23.915+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:24.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:24.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:24.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:24.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:24.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:24.808+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:24.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:24.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:25.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:24.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:25.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:24.895+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:25.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:25.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.868 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:25.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.869 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:25.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:25.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:25.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:25.869 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:25.627+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:25.877+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:25.877+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:25.878+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:25.878+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:26.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:26.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:26.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:26.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:26.875 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:26.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:26.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:26.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:26.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:26.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:26.876 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:26.581+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:26.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:26.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:26.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:26.884+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:27.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:27.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:27.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:27.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:27.776 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:27.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:28.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:27.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:28.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:27.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:28.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:27.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:28.033 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:27.866+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:28.034 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:27.785+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.034 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:27.785+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.034 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:27.785+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:28.042+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17818 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:28.885 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:28.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.885 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:28.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.885 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:28.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:28.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:28.579+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:28.885 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:28.579+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:28.886 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:28.579+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:28.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:28.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:28.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:28.895+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:29.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:29.788+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:29.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:29.788+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:29.890 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:29.788+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:29.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:29.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:29.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:29.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:29.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:29.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:30.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:29.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:29.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:29.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:30.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:29.900+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:30.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:30.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:30.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:30.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:30.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:30.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:30.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:30.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:30.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:30.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:30.861 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:30.793+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:30.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:30.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:30.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:31.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:30.871+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:31.838 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:31.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:31.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:31.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:31.839 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:31.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:31.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:31.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:31.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:31.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:31.839 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:31.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:31.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:31.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:31.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:31.848+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:32.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:32.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:32.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:32.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:32.737 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:32.569+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:32.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:32.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:32.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:32.855+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:32.747+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:33.036 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:32.747+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:33.037 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:32.747+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:33.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:33.045+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17894 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:33.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:33.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:33.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:33.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:33.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:33.539+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:33.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:33.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:33.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:33.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:33.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:33.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:34.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:33.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:34.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:33.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:34.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:33.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:34.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:33.859+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:34.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:34.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:34.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:34.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:34.827 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:34.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:34.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:34.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:34.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:34.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:34.828 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:34.796+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:34.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:34.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:34.837+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:34.837+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:36.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:35.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:36.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:35.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:36.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:35.773+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:35.610+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:35.610+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:36.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:35.610+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:35.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:35.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:35.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:36.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:35.827+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:36.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:36.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:36.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:36.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:36.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:36.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:36.850+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:36.850+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:36.850+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:37.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:36.850+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:37.713+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:37.713+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:37.713+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:37.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:37.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:37.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:37.634+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:38.054 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:38.049+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 17967 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:38.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:37.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:38.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:37.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:38.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:37.809+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:38.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:37.809+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:39.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:38.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:38.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:38.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:38.823+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:38.823+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:38.823+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:39.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:38.823+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:38.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:38.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:38.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:39.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:39.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:39.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:39.844 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:39.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:39.844 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:39.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:39.844 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:39.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:40.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:39.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:40.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:39.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:40.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:39.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:40.174 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:39.853+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:40.877 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:40.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:40.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:40.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:40.878 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:40.764+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:40.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:40.557+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:40.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:40.557+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:40.878 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:40.557+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:40.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:40.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:40.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:40.887+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:41.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:41.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:41.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:41.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:41.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:41.519+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:41.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:41.762+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:41.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:41.762+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:41.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:41.762+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:41.852+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:41.852+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:41.852+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:41.852+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:42.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:42.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:42.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:42.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:42.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:42.786+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:42.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:42.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:42.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:42.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:42.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:42.544+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:43.164 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:43.051+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18034 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:43.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:42.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:43.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:42.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:43.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:42.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:43.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:42.880+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:43.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:43.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:43.847 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:43.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:43.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:43.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:43.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:43.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:43.848 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:43.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:44.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:43.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:44.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:43.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:44.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:43.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:44.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:43.857+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:44.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:44.568+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:44.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:44.568+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:44.829 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:44.568+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:44.829 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:44.787+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:44.830 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:44.787+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:44.830 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:44.787+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:44.838+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:44.838+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:44.838+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:44.838+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:45.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:45.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:45.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:45.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:45.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:45.884+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:45.884+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:45.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:45.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:46.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:45.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:46.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:46.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:46.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:46.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:46.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:46.490+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:47.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:46.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:47.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:46.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:47.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:46.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:47.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:46.856+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:46.868+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:46.868+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:47.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:46.868+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:47.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:47.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:47.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:48.167 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:48.053+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18103 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:48.167 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:47.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:47.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:47.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:47.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:47.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:47.856+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:48.168 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:47.857+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:48.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:48.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:48.865 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:48.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:48.866 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:48.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:48.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:48.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:48.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:48.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:48.866 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:48.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:48.875+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:48.875+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:48.875+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:48.875+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:49.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:49.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:49.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:49.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:49.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:49.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:50.174 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:49.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:49.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:49.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:49.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:49.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:49.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:50.175 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:49.925+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:50.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:50.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:50.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:50.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:50.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:50.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:51.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:50.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:51.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:50.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:51.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:50.880+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:50.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:51.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:50.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:51.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:50.933+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:51.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:50.933+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:51.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:51.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:51.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:51.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:51.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:51.523+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:51.891+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:51.891+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:51.891+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:52.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:51.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:52.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:51.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:52.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:51.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:52.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:51.951+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:52.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:52.496+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:52.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:52.496+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:52.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:52.496+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:53.168 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:53.056+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18170 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:53.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:53.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:53.169 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:53.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:52.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:53.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:52.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:53.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:52.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:53.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:52.950+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:53.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:53.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:53.979 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:53.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:53.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:53.518+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:53.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:53.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:53.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:54.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:53.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:54.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:53.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:54.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:53.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:54.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:53.947+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:54.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:54.559+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:54.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:54.559+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:54.953 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:54.559+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:55.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:54.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:55.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:54.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:55.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:54.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:55.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:54.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:55.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:54.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:55.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:54.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:55.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:54.968+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:55.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:55.530+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:55.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:55.530+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:55.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:55.530+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:56.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:55.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:56.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:55.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:56.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:55.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:56.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:55.959+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:56.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:55.959+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:56.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:55.959+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:56.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:55.959+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:56.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:56.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:56.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:56.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:56.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:57.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:56.984+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:57.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:56.984+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:57.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:56.984+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:57.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:56.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:57.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:56.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:57.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:56.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:57.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:56.988+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:57.514+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:57.514+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:57.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:57.514+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:58.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:36:58.059+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18243 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:36:58.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:57.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:58.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:57.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:58.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:57.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:58.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:58.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:58.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:58.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:58.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:58.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:58.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:58.002+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:58.489+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:58.489+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:58.489+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.008+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.008+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.008+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:36:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.008+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:36:59.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:58.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:58.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:59.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:58.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:36:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:59.488+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:59.488+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:36:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:36:59.488+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:00.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:59.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:00.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:59.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:00.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:36:59.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:00.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:00.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:00.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:00.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:36:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:36:59.990+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:00.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:00.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:00.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:00.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:00.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:00.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:00.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:00.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:00.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:00.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:00.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:00.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:01.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:00.946+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:01.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:01.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:01.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:01.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:01.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:01.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:01.952+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:01.952+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:01.952+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:01.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:01.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:01.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:02.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:01.923+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:02.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:02.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:02.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:03.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:03.062+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18319 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:03.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:02.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:03.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:02.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:03.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:02.966+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:03.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:02.966+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:02.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:02.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:03.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:02.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:03.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:03.501+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:03.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:03.501+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:03.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:03.501+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:03.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:03.935+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:03.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:03.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:04.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:03.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:04.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:04.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:04.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:04.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:04.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:04.482+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:04.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:04.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:04.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:04.937+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:05.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:04.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:05.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:04.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:05.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:04.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:05.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:05.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:05.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:05.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:05.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:05.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:05.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:05.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:05.968+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:05.968+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:06.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:05.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:06.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:05.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:06.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:05.930+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:06.508+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:06.508+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:06.508+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:06.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:06.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:06.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:06.920+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:07.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:06.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:07.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:06.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:07.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:06.917+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:07.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:07.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:07.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:07.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:07.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:07.515+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:08.179 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:08.064+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18397 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:07.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:07.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:07.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:07.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:07.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:08.180 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:07.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:08.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:07.965+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:08.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:08.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:08.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:08.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:08.856 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:08.562+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:09.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:08.866+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:09.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:08.866+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:09.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:08.866+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:08.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:09.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:08.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:09.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:08.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:09.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:08.917+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:09.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:09.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:09.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:09.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:09.875 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:09.577+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:10.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:09.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:09.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:09.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:09.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:09.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:09.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:10.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:09.885+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:10.860 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:10.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:10.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:10.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:10.861 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:10.598+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:10.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:10.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:10.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:11.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:10.904+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:11.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:10.904+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:11.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:10.904+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:11.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:10.904+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:11.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:11.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:11.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:11.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:11.820 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:11.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:12.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:11.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:12.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:11.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:12.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:11.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:12.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:11.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:12.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:11.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:12.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:11.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:12.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:11.934+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:12.780 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:12.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:12.780 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:12.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:12.780 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:12.527+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:13.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:12.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:13.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:12.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:13.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:12.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:12.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:13.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:12.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:13.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:12.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:13.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:12.922+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:13.493 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:13.067+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18473 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:13.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:13.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:13.502+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:14.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:13.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:14.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:13.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:14.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:13.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:14.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:13.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:14.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:13.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:14.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:13.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:14.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:13.928+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:14.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:14.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:14.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:14.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:14.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:14.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:14.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:14.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:14.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:15.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:14.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:15.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:14.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:15.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:14.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:15.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:14.935+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:15.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:15.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:15.447+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:16.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:15.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:16.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:15.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:16.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:15.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:16.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:15.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:16.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:15.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:16.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:15.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:16.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:15.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:16.435+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:16.435+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:16.435+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:17.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:16.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:17.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:16.934+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:17.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:17.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:17.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:16.867+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:17.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:17.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:17.450+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:18.084 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:18.070+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18552 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:18.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:17.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:18.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:17.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:18.085 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:17.835+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:18.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:17.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:18.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:17.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:18.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:17.970+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:18.086 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:17.970+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:18.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:18.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:18.493+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:19.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:18.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:19.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:18.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:19.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:18.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:19.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:19.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:19.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:19.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.005+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:19.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:19.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:19.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:19.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:19.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:19.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:20.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:19.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:20.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:19.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:20.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:19.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:20.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:20.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:19.963+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:20.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:20.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:20.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:20.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:20.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:20.969+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:21.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:20.969+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:21.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:20.969+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:20.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:20.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:20.969+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:21.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:20.969+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:21.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:21.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:21.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:21.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:21.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:21.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:22.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:21.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:22.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:21.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:22.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:21.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:22.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:21.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:22.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:21.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:22.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:21.989+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:22.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:21.989+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:22.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:22.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:22.476+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:23.291 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:23.073+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18628 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:22.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:22.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:22.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:23.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:23.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:23.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:23.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:23.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:23.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:23.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:23.024+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:23.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:23.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:23.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:24.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:23.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:24.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:23.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:24.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:23.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:24.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:24.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:24.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:24.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:24.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:24.023+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:24.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:24.023+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:24.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:24.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:24.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:24.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:24.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:24.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:25.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:24.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:25.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:24.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:25.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:24.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:25.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:25.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:25.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:25.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:25.053+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:25.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:25.053+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:25.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:25.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:25.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:26.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:25.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:26.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:25.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:26.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:25.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:26.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:26.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:26.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:26.102+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:26.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:26.102+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:26.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:26.508+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:26.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:26.508+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:26.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:26.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:27.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:27.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:27.145+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:27.145+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:26.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:26.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:27.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:26.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:27.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:27.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:27.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:27.558+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:28.158 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:28.076+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18706 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:28.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:27.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:28.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:27.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:28.159 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:27.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:28.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:28.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:28.168+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:28.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:28.169+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:28.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:28.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:28.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:28.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:28.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:28.600+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:29.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:29.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:29.158+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:29.158+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:28.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:28.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:29.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:28.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:29.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:29.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:29.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:29.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:29.910 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:29.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:30.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:29.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:30.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:29.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:30.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:29.919+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:30.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:30.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:30.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:30.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:30.178 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:30.162+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:30.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:30.162+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:30.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:30.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:30.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:30.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:30.913 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:30.564+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:31.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:30.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:31.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:30.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:31.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:30.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:31.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:31.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:31.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:31.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:31.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:31.127+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:31.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:31.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:31.915 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:31.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:31.916 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:31.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:32.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:31.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:32.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:31.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:32.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:31.925+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:32.166+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:32.166+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:32.166+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:32.167+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:32.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:32.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:32.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:32.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:32.922 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:32.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:33.179 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:33.079+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18782 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:33.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:32.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:33.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:32.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:33.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:32.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:33.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:33.189+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:33.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:33.189+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:33.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:33.189+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:33.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:33.189+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:33.929 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:33.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:33.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:33.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:33.930 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:33.653+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:34.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:34.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:34.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:33.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:34.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:34.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:34.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:34.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:34.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:34.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:34.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:34.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:34.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:34.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:34.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:34.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:34.941 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:34.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:35.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:34.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:35.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:34.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:35.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:34.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:35.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:35.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:35.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:35.179+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:35.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:35.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:35.965 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:35.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:35.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:35.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:35.966 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:35.718+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:36.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:35.975+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:35.975+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:36.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:35.975+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:36.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:36.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:36.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:36.229+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:36.975 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:36.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:36.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:36.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:36.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:36.717+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:37.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:36.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:37.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:36.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:37.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:36.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:37.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:37.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:37.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:37.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:37.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:37.185+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:37.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:37.185+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:38.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:37.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:38.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:37.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:38.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:37.698+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:38.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:38.082+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18860 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:38.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:38.021+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:38.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:38.021+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:38.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:38.021+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:38.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:38.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:38.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:38.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:38.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:38.236+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:38.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:38.236+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:39.017 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:38.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:39.018 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:38.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:39.018 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:38.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.026+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:39.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:39.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:39.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:39.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:39.196+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:39.979 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:39.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:39.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:39.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:39.980 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:39.685+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:40.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:40.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:40.294 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:39.989+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:40.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:40.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:40.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:40.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:40.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:40.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:40.294 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:40.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:40.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:40.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:40.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:40.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:40.949 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:40.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:41.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:40.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:41.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:40.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:41.273 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:40.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:41.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:41.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:41.282+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:41.282+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:42.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:41.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:42.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:41.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:42.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:41.946+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:42.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:41.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:42.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:41.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:42.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:41.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:42.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:42.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:42.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:42.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:42.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:42.290+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:42.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:42.290+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:43.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:42.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:43.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:42.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:43.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:42.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:43.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:42.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:43.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:42.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:43.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:42.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:43.542 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:43.085+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 18937 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:43.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:43.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:43.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:43.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:43.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:43.266+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:43.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:43.266+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:43.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:43.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:43.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:44.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:43.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:44.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:43.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:44.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:43.768+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:44.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:44.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:44.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:44.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:44.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:44.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:44.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:44.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:44.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:44.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:44.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:45.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:44.723+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:45.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:44.723+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:45.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:44.723+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:45.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:45.275+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:45.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:45.275+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:45.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:45.275+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:45.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:45.275+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:45.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:45.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:45.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:45.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:45.955 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:45.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:46.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:45.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:45.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:45.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:46.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:46.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:46.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:46.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:46.267+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:46.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:46.267+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:46.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:46.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:46.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.967 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:46.976+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:46.967 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:46.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:46.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:46.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:46.968 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:46.727+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:47.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:47.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:47.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:47.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:47.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:47.254+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:47.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:47.254+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:48.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:47.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:48.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:47.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:48.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:47.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:48.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:47.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:48.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:47.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:48.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:47.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:48.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:48.088+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19014 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:48.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:48.272+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:48.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:48.272+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:48.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:48.272+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:48.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:48.272+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:49.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:48.977+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:49.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:48.977+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:49.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:48.977+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:48.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:48.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:49.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:48.732+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:49.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:49.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:49.745 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:49.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:49.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:49.304+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:49.746 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:49.304+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:50.000+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:50.000+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:50.000+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:50.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:49.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:50.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:49.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:50.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:49.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:50.348+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:50.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:50.349+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:50.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:50.349+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:50.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:50.349+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:50.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:50.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:50.784+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:51.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:51.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:51.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:51.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:51.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:51.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:51.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:51.355+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:51.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:51.355+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:51.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:51.355+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:51.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:51.355+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:52.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:51.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:52.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:51.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:52.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:51.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:52.050+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:52.050+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:52.050+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:52.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:52.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:52.328+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:52.328+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:53.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:53.041+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:53.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:53.041+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:53.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:53.041+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:53.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:52.800+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:53.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:52.800+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:53.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:52.800+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:53.333 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:53.090+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19091 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:53.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:53.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:53.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:53.343+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:54.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:53.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:54.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:53.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:54.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:53.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:54.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:54.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:54.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:54.005+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:54.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:54.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:54.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:54.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:54.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:54.380+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:54.767 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:54.380+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:55.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:54.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:54.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:54.777+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:55.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:55.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:55.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:55.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:55.397 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:55.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:55.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:55.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:55.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:55.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:55.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:55.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:55.730 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:55.406+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:56.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:56.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:56.093+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:56.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:56.093+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:56.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:56.093+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:56.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:56.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:56.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:56.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:57.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:57.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:57.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:57.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:57.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:57.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:57.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:57.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:57.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:57.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:57.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:57.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:57.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:57.454+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:58.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:57.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:58.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:57.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:58.084 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:57.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:58.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:37:58.093+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19169 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:37:58.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:58.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:58.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:58.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:58.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:58.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:58.499+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:37:59.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:58.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:59.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:58.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:59.173 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:58.850+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:37:59.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:59.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:59.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:59.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:59.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:37:59.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:37:59.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:59.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:59.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:59.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:37:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:37:59.546+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:00.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:59.868+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:00.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:59.868+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:00.170 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:37:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:37:59.868+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:00.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:00.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:00.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:00.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:00.524 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:00.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:00.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:00.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:00.534+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:00.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:00.534+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:01.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:00.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:01.159 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:00.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:01.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:00.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:01.443 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:01.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:01.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:01.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:01.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:01.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:01.707 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:01.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:01.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:01.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:01.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:01.553+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:01.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:01.553+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:02.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:01.830+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:02.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:01.830+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:02.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:01.830+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:02.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:02.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:02.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:02.129+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:02.824 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:02.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:02.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:02.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:02.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:02.571+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:02.825 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:02.571+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:03.087 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:02.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:03.087 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:02.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:03.087 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:02.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:03.541 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:03.096+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19246 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:03.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:03.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:03.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:03.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:03.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:03.150+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:03.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:03.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:03.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:03.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:03.852 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:03.562+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:03.853 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:03.562+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:04.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:04.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:04.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:04.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:03.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:04.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:03.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:04.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:03.862+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:04.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:04.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:04.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:04.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:04.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:04.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:04.564+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:05.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:04.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:05.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:04.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:05.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:04.847+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:05.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:05.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:05.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:05.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:05.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:05.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:05.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:05.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:05.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:05.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:05.813 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:05.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:05.814 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:05.567+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:06.093 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:05.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:06.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:05.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:06.094 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:05.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:06.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:06.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:06.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:06.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:06.532 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:06.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:06.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:06.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:06.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:06.542+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:07.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:06.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:07.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:06.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:07.130 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:06.846+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:07.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:07.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:07.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:07.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:07.503 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:07.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:07.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:07.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:07.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:07.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:07.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:07.549+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:07.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:07.549+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:08.111 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:08.099+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19323 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:08.111 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:08.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:07.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:08.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:07.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:08.112 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:07.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:08.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:08.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:08.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:08.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:08.518+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:09.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:08.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:09.090 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:08.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:09.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:08.841+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:09.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:09.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:09.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:09.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:09.465 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:09.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:09.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:09.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:09.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:09.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:10.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:10.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:10.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:10.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:10.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:10.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:10.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:09.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:10.181 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:09.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:10.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:09.848+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:10.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:10.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:10.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:10.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:10.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:11.082+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:11.082+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:11.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:11.082+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:11.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:10.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:11.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:10.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:11.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:10.843+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:11.708 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:11.455+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:11.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:11.455+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:11.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:11.455+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:11.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:11.455+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:12.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:11.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:11.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:12.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:11.873+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:12.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:12.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:12.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:12.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:12.453 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:12.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:12.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:12.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:12.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:12.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:12.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:12.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:12.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:12.464+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:13.106 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:13.102+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19400 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:12.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:12.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:12.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:13.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:13.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:13.107 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:13.058+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:13.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:13.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:13.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:13.731 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:13.421+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:14.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:13.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:14.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:13.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:14.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:13.863+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:14.388 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:14.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:14.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:14.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:14.389 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:14.051+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:14.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:14.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:14.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:14.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:14.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:14.399+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:14.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:14.399+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:15.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:15.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:15.061+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:15.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:14.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:14.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:15.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:14.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:15.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:15.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:15.416+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:15.416+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:16.073 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:15.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:16.073 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:15.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:16.073 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:15.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:16.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:16.083+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:16.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:16.083+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:16.421 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:16.083+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:16.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:16.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:16.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:16.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:16.432+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:16.432+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:17.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:16.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:17.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:16.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:17.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:16.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:17.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:17.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:17.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:17.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:17.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:17.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:17.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:17.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:17.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:17.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:17.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:17.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:17.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:17.465+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:18.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:17.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:18.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:17.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:18.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:17.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:18.410 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:18.105+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19478 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:18.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:18.094+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:18.410 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:18.094+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:18.411 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:18.094+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:18.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:18.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:18.420+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:18.420+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:19.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:18.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:19.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:18.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:19.125 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:18.804+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:19.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:19.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:19.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:19.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:19.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:19.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:19.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:19.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:19.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:19.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:19.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:19.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:19.777 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:19.411+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:19.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:19.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:20.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:19.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:20.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:20.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:20.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:20.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:20.407 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:20.153+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:20.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:20.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:20.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:20.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:20.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:20.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:21.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:20.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:21.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:20.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:21.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:20.757+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:21.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:21.109+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:21.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:21.109+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:21.404 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:21.109+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:21.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:21.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:21.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:21.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:21.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:21.413+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:21.709 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:21.414+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:22.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:21.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:21.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:21.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:22.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:22.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:22.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:22.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:22.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:22.152+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:22.405+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:22.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:22.405+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:23.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:22.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:23.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:22.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:23.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:22.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:23.353 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:23.108+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19555 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:23.353 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:23.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:23.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:23.354 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:23.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:23.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:23.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:23.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:23.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:23.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:23.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:23.667 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:23.363+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:24.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:23.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:24.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:23.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:24.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:23.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:24.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:24.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:24.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:24.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:24.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:24.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:24.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:24.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:24.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:24.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:24.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:24.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:24.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:24.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:24.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:24.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:24.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:25.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:25.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:25.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:25.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:25.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:25.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:25.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:25.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:25.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:25.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:25.419+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:25.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:25.420+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:26.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:25.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:26.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:25.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:26.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:25.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:26.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:26.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:26.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:26.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:26.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:26.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:26.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:26.434+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:26.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:26.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:26.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:26.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:26.238+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:27.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:26.693+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:26.693+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:26.693+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:27.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:27.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:27.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:27.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:27.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:27.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:27.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:27.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:27.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:27.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:27.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:27.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:27.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:27.428+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:28.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:27.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:28.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:27.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:28.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:27.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:28.444 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:28.111+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19633 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:28.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:28.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:28.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:28.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:28.444 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:28.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:28.719 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:28.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:28.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:28.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:28.720 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:28.454+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:29.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:28.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:29.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:28.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:29.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:28.729+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:29.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:29.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:29.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:29.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:29.749 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:29.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:29.749 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:29.409+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:29.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:29.409+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:29.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:29.409+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:29.750 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:29.409+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:30.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:29.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:30.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:29.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:30.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:29.759+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:30.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:30.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:30.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:30.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:30.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:30.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:30.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:30.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:30.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:30.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:30.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:30.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:30.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:30.412+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:31.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:30.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:31.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:30.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:31.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:30.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:31.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:31.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:31.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:31.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:31.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:31.286+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:31.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:31.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:31.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:31.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:31.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:31.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:31.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:31.428+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:32.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:31.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:32.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:31.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:32.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:31.811+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:32.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:32.264+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:32.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:32.264+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:32.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:32.264+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:32.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:32.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:32.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:32.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:32.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:32.396+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:32.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:32.396+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:33.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:32.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:33.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:32.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:33.104 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:32.861+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:33.394 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:33.113+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19710 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:33.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:33.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:33.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:33.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:33.395 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:33.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:33.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:33.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:33.404+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:33.404+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:34.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:33.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:34.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:33.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:34.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:33.891+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:34.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:34.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:34.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:34.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:34.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:34.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:34.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:34.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:34.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:34.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:34.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:34.441+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:34.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:35.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:34.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:35.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:34.845+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:35.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:35.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:35.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:35.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:35.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:35.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:35.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:35.444+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:35.304+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:35.304+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:35.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:35.304+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:36.281 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:35.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:36.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:35.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:36.282 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:35.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:36.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:36.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:36.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:36.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:36.292+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:36.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:36.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:36.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:36.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:36.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:36.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:36.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:37.284 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:36.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:37.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:36.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:37.285 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:36.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:37.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:37.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:37.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:37.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:37.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:37.295+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:37.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:37.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:37.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:37.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:37.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:37.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:37.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:37.465+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:38.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:37.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:38.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:37.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:38.107 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:37.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:38.475 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:38.116+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19788 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:38.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:38.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:38.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:38.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:38.476 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:38.332+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:38.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:38.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:38.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:38.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:38.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:39.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:38.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:39.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:38.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:39.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:38.802+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:39.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:39.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:39.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:39.359+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:39.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:39.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:39.530+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:39.530+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:40.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:39.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:40.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:39.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:40.182 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:39.801+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:40.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:40.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:40.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:40.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:40.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:40.385+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:40.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:40.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:40.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:40.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:40.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:40.525+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:40.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:40.525+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:41.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:40.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:41.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:40.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:41.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:40.851+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:41.711 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:41.341+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:41.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:41.341+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:41.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:41.341+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:41.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:41.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:41.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:41.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:41.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:41.516+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:41.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:41.516+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:42.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:41.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:41.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:41.805+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:42.784 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:42.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:42.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:42.785 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:42.367+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:42.785 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:42.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:42.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:42.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:42.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:42.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:42.786 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:42.472+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:43.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:42.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:43.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:42.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:43.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:42.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:43.360 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:43.119+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19865 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:43.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:43.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:43.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:43.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:43.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:43.370+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:43.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:43.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:43.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:43.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:43.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:43.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:43.776 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:43.481+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:44.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:43.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:44.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:43.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:44.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:43.785+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:44.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:44.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:44.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:44.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:44.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:44.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:44.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:45.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:44.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:45.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:44.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:45.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:44.824+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:45.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:45.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:45.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:45.387+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:45.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:45.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:45.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:46.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:46.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:46.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:45.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:46.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:46.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:46.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:46.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:46.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:46.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:46.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:46.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:46.445+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:46.445+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:47.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:46.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:47.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:46.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:47.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:46.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:47.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:47.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:47.319+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:47.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:47.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:47.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:47.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:48.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:47.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:48.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:47.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:48.113 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:47.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:48.488 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:48.122+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 19942 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:48.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:48.341+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:48.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:48.341+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:48.489 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:48.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:48.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:48.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:48.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:48.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:48.499+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:49.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:48.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:49.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:48.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:49.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:48.803+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:49.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:49.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:49.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:49.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:49.532+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:49.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:49.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:49.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:49.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:49.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:49.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:50.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:49.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:50.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:49.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:50.183 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:49.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:50.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:50.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:50.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:50.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:50.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:50.300+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:51.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:50.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:51.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:50.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:51.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:50.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:50.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:50.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:50.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:50.572+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:51.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:51.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:51.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:51.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:51.712 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:51.336+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:51.712 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:51.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:51.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:51.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:51.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:51.547+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:52.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:51.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:51.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:52.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:51.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:52.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:52.376+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:52.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:52.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:52.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:52.377+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:53.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:52.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:53.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:52.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:53.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:52.796+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:52.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:52.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:52.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:52.554+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:53.388 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:53.125+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20019 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:53.667 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:53.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:53.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:53.399+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:53.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:53.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:53.594+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:53.668 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:53.594+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:54.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:53.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:54.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:53.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:54.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:53.755+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:54.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:54.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:54.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:54.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:54.742 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:54.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:54.743 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:54.551+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:54.743 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:54.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:54.743 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:54.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:54.743 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:54.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:55.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:55.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:55.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:55.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:55.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:55.577+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:55.748 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:55.577+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:55.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:55.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:55.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:55.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:55.748 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:55.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:56.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:55.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:55.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:56.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:55.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:56.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:56.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:56.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:56.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:56.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:56.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:56.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:56.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:56.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:56.611+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:57.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:57.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:56.779+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:57.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:57.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:57.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:57.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:57.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:57.389+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:57.787 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:57.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:57.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:57.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:57.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:57.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:57.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:57.567+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:58.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:57.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:58.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:57.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:58.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:57.797+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:58.520 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:38:58.128+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20097 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:38:58.520 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:58.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:58.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:58.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:58.521 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:58.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:58.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:58.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:58.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:58.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:58.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:58.530+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:38:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:58.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:58.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:58.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:58.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:58.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:38:59.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:59.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:59.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:59.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:59.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:38:59.364+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:38:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:59.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:59.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:59.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:59.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:38:59.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:38:59.522+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:00.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:59.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:00.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:59.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:00.184 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:38:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:38:59.810+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:00.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:00.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:00.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:00.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:01.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:00.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:00.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:00.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:00.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:00.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:00.565+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:00.565+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:01.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:01.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:01.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:01.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:01.713 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:01.414+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:01.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:01.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.713 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:01.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:01.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:01.714 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:01.611+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:02.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:01.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:02.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:01.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:02.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:01.829+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:02.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:02.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:02.403+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:02.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:02.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:02.771 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:02.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:02.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:02.614+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:02.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:03.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:02.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:03.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:02.869+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:03.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:03.130+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20174 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:03.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:03.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:03.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:03.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:03.578 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:03.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:03.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:03.588+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:03.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:03.588+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:03.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:03.588+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:03.902 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:03.588+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:04.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:03.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:04.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:03.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:04.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:03.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:04.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:04.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:04.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:04.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:04.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:04.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:04.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:04.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:04.938 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:04.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:04.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:04.597+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:04.939 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:04.597+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:05.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:04.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:05.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:04.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:05.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:04.949+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:05.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:05.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:05.404+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:05.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:05.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:05.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:05.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:05.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:05.626+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:06.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:05.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:06.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:05.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:06.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:05.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:06.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:06.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:06.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:06.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:06.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:06.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:06.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:06.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:06.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:06.611+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:06.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:06.611+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:07.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:06.939+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:07.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:06.939+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:07.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:06.939+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:07.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:07.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:07.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:07.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:07.616 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:07.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:07.889 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:07.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:07.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:07.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:07.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:07.626+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:07.890 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:07.626+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:08.143 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:08.133+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20251 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:08.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:07.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:08.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:07.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:08.144 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:07.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:08.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:08.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:08.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:08.476+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:08.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:08.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:08.667+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:08.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:08.667+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:09.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:08.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:09.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:08.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:09.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:08.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:09.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:09.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:09.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:09.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:09.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:09.464+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:09.672+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:09.672+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:09.672+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:09.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:09.673+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:10.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:09.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:10.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:09.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:10.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:09.872+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:10.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:10.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:10.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:10.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:10.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:10.686+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:10.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:10.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:10.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:10.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:10.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:10.466+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:11.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:10.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:11.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:10.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:11.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:10.835+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:11.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:11.726+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:11.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:11.726+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:11.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:11.726+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:11.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:11.726+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:11.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:11.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:11.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:11.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:11.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:11.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:12.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:11.828+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:12.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:11.828+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:12.099 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:11.828+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:12.771 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:12.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:12.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:12.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:12.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:12.460+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:12.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:12.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:12.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:12.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:12.772 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:12.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:12.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:12.683+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:13.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:12.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:13.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:12.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:13.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:12.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:13.471 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:13.136+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20328 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:13.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:13.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:13.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:13.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:13.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:13.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:13.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:13.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:13.684+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:14.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:13.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:14.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:13.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:14.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:13.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:14.654 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:14.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:14.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:14.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:14.655 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:14.498+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:14.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:14.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:14.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:14.664+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:14.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:14.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:15.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:14.731+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:15.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:15.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:15.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:15.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:15.716 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:15.455+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:15.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:15.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.716 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:15.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:15.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:15.717 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:15.642+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:16.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:15.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:15.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:15.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:16.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:16.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:16.722+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:16.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:16.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:16.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:16.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:16.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:16.486+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:16.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:16.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:16.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:16.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:16.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:16.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:16.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:16.668+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:17.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:17.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:17.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:17.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:17.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:17.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:17.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:17.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:17.518+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:17.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:17.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:17.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:17.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:17.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:17.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:17.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:17.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:18.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:18.139+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20406 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:18.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:18.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:18.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:18.470+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:18.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:18.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:18.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:18.657+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:18.657+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:18.657+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:18.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:18.657+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:19.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:19.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:19.477+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:19.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:19.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:19.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:19.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:19.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:19.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:19.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:19.794 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:19.619+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:20.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:20.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:20.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:20.500+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:20.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:20.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:20.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:20.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:20.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:20.592+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:20.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:20.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:20.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:20.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:20.643+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:21.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:21.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:21.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:21.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:21.617 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:21.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:22.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:21.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:21.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:21.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:21.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:21.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:21.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:21.627+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:22.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:22.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:22.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:22.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:22.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:22.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:22.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:22.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:22.629+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:22.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:22.629+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:23.493 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:23.142+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20483 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:23.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:23.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:23.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:23.503+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:23.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:23.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:23.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:23.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:23.637+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:23.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:23.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:23.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:23.673+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:24.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:24.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:24.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:24.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:24.653 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:24.532+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:24.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:24.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:25.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:24.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:24.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:24.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:24.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:24.663+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:26.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:25.675+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:26.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:25.675+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:26.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:25.675+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:25.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:25.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:25.571+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:25.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:25.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:25.651+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:26.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:25.651+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:27.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:26.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:26.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:27.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:26.665+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:26.553+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:26.553+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:26.553+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:26.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:26.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:26.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:26.658+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:27.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:27.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:27.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:27.583+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:27.673+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:27.673+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:27.673+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:27.673+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:27.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:27.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:27.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:27.639+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:28.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:28.145+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20559 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:29.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:28.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:29.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:28.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:29.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:28.648+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:29.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:28.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:29.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:28.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:29.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:28.623+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:28.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:28.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:28.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:29.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:28.682+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:30.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:29.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:29.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:29.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:29.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:29.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:29.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:29.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:29.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:29.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:29.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:30.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:30.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:30.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:30.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:30.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:30.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:30.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:30.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:30.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:30.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:30.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:30.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:30.645+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:31.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:32.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:31.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:32.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:31.712+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:32.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:31.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:32.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:31.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:32.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:31.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:31.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:31.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:31.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:31.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:33.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:32.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:32.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:32.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:33.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:32.621+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:33.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:32.621+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:33.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:32.621+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:33.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:32.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:33.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:32.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:33.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:32.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:33.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:32.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:33.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:33.148+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20635 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:33.760+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:34.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:33.760+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:34.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:33.760+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:33.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:33.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:33.607+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:33.712+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:33.712+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:33.712+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:34.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:33.712+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:35.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:34.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:34.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:34.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:34.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:34.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:34.571+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:34.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:34.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:34.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:34.683+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:35.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:35.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:35.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:35.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:35.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:35.536+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:35.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:35.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:35.643+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:35.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:35.643+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:36.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:36.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:36.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:36.570+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:36.570+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.043 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:36.570+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:36.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:36.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:36.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.044 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:36.603+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:37.535+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:37.535+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:37.535+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:37.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:37.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:37.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:37.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:37.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:37.637+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:38.519 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:38.152+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20713 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:38.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:38.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:38.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:38.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:38.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:38.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:38.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:38.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:38.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:38.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:38.792 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:38.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:38.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:38.652+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:38.793 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:38.652+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:40.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:39.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:40.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:39.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:40.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:39.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:40.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:39.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:40.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:39.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:40.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:39.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:40.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:39.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:40.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:39.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:40.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:39.630+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:40.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:39.630+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:40.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:41.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:40.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:41.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:40.706+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:40.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:40.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:41.042 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:40.556+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:41.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:40.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:40.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:40.668+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:41.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:40.668+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:42.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:41.587+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:41.587+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:41.587+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:41.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:41.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:41.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:41.705+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:41.705+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:41.705+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:41.705+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:42.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:42.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:42.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:42.696+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:42.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:42.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:42.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:42.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:42.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:42.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:42.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:42.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:42.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:42.693+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:43.518 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:43.155+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20790 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:43.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:43.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:43.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:43.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:43.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:43.703+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:43.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:43.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:43.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:43.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:43.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:43.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:44.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:44.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:44.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:44.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:44.666+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:44.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:44.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:44.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:44.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:44.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:44.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:44.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:44.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:44.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:44.685+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:45.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:45.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:45.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:45.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:45.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:45.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:45.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:45.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:45.624+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:45.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:45.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:45.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:45.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:45.702+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:46.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:46.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:46.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:46.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:46.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:46.529+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:46.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:46.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:46.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:46.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:46.792 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:46.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:46.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:46.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:46.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:46.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:46.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:47.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:47.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:47.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:47.674+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:47.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:47.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:47.531+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:47.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:47.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:47.693+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:47.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:47.693+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:48.500 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:48.157+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20867 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:48.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:48.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:48.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:48.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:48.511+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:48.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:48.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:48.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:48.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:48.727+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:48.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:48.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:48.793 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:48.665+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:49.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:49.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:49.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:49.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:49.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:49.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:49.749+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:49.749+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:49.749+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:49.749+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:50.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:50.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:50.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:50.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:50.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:50.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:50.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:50.652+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:50.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:50.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:50.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:50.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:50.735+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:51.754 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:51.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:51.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:51.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:51.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:51.645+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:51.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:51.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:51.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:51.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:51.755 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:51.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:52.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:51.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:52.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:51.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:52.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:51.765+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:52.035 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:51.765+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:52.787 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:52.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:52.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:52.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:52.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:52.688+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:52.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:52.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:52.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:52.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:52.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:52.484+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:53.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:52.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:53.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:52.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:52.798+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:52.798+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:53.501 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:53.161+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 20944 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:53.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:53.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:53.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:53.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:53.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:53.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:53.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:53.512+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:53.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:53.512+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:53.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:53.512+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:54.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:53.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:54.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:53.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:54.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:53.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:54.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:53.826+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:54.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:54.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:54.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:54.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:54.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:54.695+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:54.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:54.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:54.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:54.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:54.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:54.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:55.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:54.822+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:55.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:54.822+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:55.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:54.822+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:55.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:54.822+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:55.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:55.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:55.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:55.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:55.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:55.680+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:55.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:55.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:55.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:55.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:55.770 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:55.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:56.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:55.780+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:56.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:55.781+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:56.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:55.781+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:55.781+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:56.757 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:56.728+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:56.757 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:56.728+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:56.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:56.728+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:56.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:56.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:56.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:56.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:56.758 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:56.509+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:56.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:56.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:56.767+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:57.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:56.767+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:57.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:57.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:57.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:57.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:57.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:57.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:57.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:57.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:57.492+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:57.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:57.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:58.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:57.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:58.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:57.800+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:58.439 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:39:58.164+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21031 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:39:58.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:58.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:58.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:58.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:58.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:58.685+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:58.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:58.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:58.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:58.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:58.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:58.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:58.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:39:59.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:58.821+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:39:59.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:59.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:59.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:59.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:59.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:39:59.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:39:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:59.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:59.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:39:59.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:39:59.467+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:00.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:59.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:00.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:59.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:00.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:59.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:00.188 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:39:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:39:59.855+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:00.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:00.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:00.440+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:00.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:01.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:00.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:01.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:00.672+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:00.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:00.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:00.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:01.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:00.872+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:01.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:01.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:01.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:01.474+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:01.658+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:01.658+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:01.658+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:01.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:01.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:01.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:01.887+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:02.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:02.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:02.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:02.648+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:02.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:02.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:02.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:02.464+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:03.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:02.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:03.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:02.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:03.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:02.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:03.157 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:02.910+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:03.413 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:03.167+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21108 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:03.673 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:03.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:03.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:03.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:03.674 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:03.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:04.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:03.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:04.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:03.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:03.684+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:03.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:03.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:03.887+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:03.887+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:04.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:04.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:04.645 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:04.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:04.646 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:04.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:04.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:04.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:04.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:04.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:04.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:04.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:05.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:04.913+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:04.913+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:04.913+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:04.913+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:05.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:05.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:05.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:05.625+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:05.625+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:05.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:05.625+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:06.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:05.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:05.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:05.885+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:05.885+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:06.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:06.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:06.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:06.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:06.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:06.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:06.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:06.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:06.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:06.483+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:07.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:06.890+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:06.890+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:06.890+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:06.890+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:07.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:07.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:07.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:07.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:07.550 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:07.473+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:07.927 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:07.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:07.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:07.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:07.928 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:07.560+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:08.182 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:08.170+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21185 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:08.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:07.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:08.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:07.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:08.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:07.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:08.183 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:07.938+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:08.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:08.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:08.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:08.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:08.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:08.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:08.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:08.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:08.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:09.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:08.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:08.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:08.910+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:08.910+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:09.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:09.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:09.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:09.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:09.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:09.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:09.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:09.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:09.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:09.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:09.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:09.564+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:10.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:09.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:10.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:09.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:10.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:09.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:10.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:09.950+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:10.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:10.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:10.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:10.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:10.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:10.468+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:10.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:10.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:10.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:10.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:10.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:10.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:11.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:10.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:11.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:10.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:11.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:10.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:11.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:10.948+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:11.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:11.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:11.449+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:11.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:11.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:11.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:11.637+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:11.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:11.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:11.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:11.920+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:12.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:12.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:12.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:12.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:12.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:12.430+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:12.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:12.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:12.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:12.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:12.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:12.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:13.163 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:12.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:13.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:12.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:13.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:12.902+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:13.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:12.902+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:13.459 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:13.173+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21262 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:13.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:13.469+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:13.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:13.469+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:13.469+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:13.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:13.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:13.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:13.599+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:14.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:13.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:14.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:13.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:14.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:13.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:13.880+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:14.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:14.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:14.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:14.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:14.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:14.471+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:14.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:14.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:14.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:14.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:14.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:14.627+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:14.896+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:14.896+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:14.896+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:14.896+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:15.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:15.580+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:15.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:15.580+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:15.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:15.580+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:15.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:15.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:15.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:15.940+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:15.940+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:16.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:15.940+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:16.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:15.940+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:16.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:16.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:16.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:16.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:16.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:16.575+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:16.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:16.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:16.427+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:17.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:16.927+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:16.927+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:16.927+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:16.927+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:17.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:17.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:17.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:17.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:17.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:17.619+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:17.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:17.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:17.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:17.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:18.178 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:18.177+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21340 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:17.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:17.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:17.937+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:18.179 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:17.937+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:18.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:18.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:18.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:18.478+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:18.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:18.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:18.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:18.595+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:19.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:18.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:19.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:18.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:19.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:18.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:19.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:18.973+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:19.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:19.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:19.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:19.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:19.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:19.448+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:19.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:19.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:19.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:19.549+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:20.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:20.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:20.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:19.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:20.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:19.996+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:20.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:20.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:20.411+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:20.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:20.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:20.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:20.508+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:21.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:21.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:21.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:21.002+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:21.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:21.003+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:21.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:21.506+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:21.719 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:21.506+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:21.720 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:21.506+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:21.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:21.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:21.720 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:21.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:22.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:22.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:22.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:22.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:22.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:22.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:22.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:22.006+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:22.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:22.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:22.755 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:22.511+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:22.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:22.424+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:22.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:22.424+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:22.756 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:22.424+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:23.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:23.180+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21417 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:23.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:23.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:23.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:23.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.016+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:23.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:23.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:23.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:23.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:23.378+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:23.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:23.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:23.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:23.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:24.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:24.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:24.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:24.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:23.997+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:24.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:24.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:24.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:24.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:24.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:24.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:24.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:24.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:24.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:24.578+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:25.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:24.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:25.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:24.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:25.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:24.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:24.947+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:25.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:25.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:25.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:25.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:25.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:25.590+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:25.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:25.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:25.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:25.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:26.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:25.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:26.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:25.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:26.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:25.975+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:26.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:25.975+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:26.383+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:26.383+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:26.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:26.383+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:26.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:26.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:26.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:26.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:26.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:26.608+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:27.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:26.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:27.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:26.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:27.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:26.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:26.943+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:27.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:27.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:27.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:27.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:27.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:27.366+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:27.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:27.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:27.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:27.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:27.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:27.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:28.290 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:28.183+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21494 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:27.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:27.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:27.962+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:28.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:27.962+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:28.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:28.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:28.601 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:28.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:28.602 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:28.386+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:28.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:28.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:28.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:28.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:28.925 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:28.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:29.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:28.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:29.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:28.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:29.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:28.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:29.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:28.935+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:29.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:29.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:29.642 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:29.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:29.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:29.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:29.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:29.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:29.934 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:29.651+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:30.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:29.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:30.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:29.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:30.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:29.945+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:30.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:29.945+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:30.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:30.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:30.351+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:30.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:30.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:30.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:30.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:30.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:30.612+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:31.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:30.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:31.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:30.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:31.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:30.923+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:31.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:30.923+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:31.574 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:31.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:31.574 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:31.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:31.574 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:31.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:31.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:31.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:31.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:31.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:31.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:31.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:32.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:31.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:31.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:31.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:32.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:31.914+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:32.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:32.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:32.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:32.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:32.578 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:32.333+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:32.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:32.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:32.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:32.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:32.913 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:32.589+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:33.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:32.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:33.176 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:32.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:33.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:32.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:33.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:32.924+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:33.541 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:33.186+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21571 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:33.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:33.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:33.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:33.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:33.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:33.321+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:33.911 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:33.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:33.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:33.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:33.912 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:33.558+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:34.279 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:33.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:34.279 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:33.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:34.279 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:33.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:34.280 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:33.922+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:34.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:34.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:34.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:34.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:34.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:34.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:34.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:34.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:34.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:34.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:34.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:34.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:34.948+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:35.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:35.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:35.541+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:35.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:35.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:35.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:35.320+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:36.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:35.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:36.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:35.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:36.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:35.976+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:36.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:35.976+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:36.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:36.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:36.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:36.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:36.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:36.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:36.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:36.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:36.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:36.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:36.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:36.507+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:37.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:37.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:37.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.016+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:37.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.016+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:37.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:37.272+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:37.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:37.272+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:37.524 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:37.272+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:37.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:37.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:37.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:37.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:37.534+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:38.238 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:38.189+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21649 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:38.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:38.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:38.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:38.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:37.971+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:38.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:38.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:38.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:38.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:38.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:38.249+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:38.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:38.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:38.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:38.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:38.915 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:38.561+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:39.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:38.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:39.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:38.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:39.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:38.926+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:39.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:38.926+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:39.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:39.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:39.207+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:39.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:39.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:39.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:39.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:39.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:39.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:40.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:39.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:40.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:39.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:40.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:39.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:40.191 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:39.941+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:40.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:40.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:40.232+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:40.970 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:40.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:40.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:40.971 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:40.588+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:41.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:41.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:41.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:41.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:41.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:41.196+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:40.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:40.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:40.981+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:40.981+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:42.003 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:41.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:42.004 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:41.604+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:42.004 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:41.605+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:42.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:42.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.014+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:42.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.014+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:42.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:42.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:42.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:42.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:42.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:42.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:42.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:42.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:42.955 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:42.586+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:43.204 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:43.193+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21726 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.965+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:42.965+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:43.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:43.205 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:43.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:43.206 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:43.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:43.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:43.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:43.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:43.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:43.924 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:43.611+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:44.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:44.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:44.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:44.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:44.259 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:44.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:44.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:43.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:44.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:43.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:44.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:43.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:44.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:43.935+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:44.935 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:44.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:44.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:44.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:44.936 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:44.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:45.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:45.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:45.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:45.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:45.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:45.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:44.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:44.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:44.947+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:45.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:44.947+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:45.931 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:45.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:45.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:45.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:45.932 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:45.643+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:45.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:46.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:45.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:46.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:45.942+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:46.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:45.942+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:46.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:46.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:46.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:47.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:46.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:47.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:46.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:47.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:46.676+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:47.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:46.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:47.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:46.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:47.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:46.925+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:47.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:46.925+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:47.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:47.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:47.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:47.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:47.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:47.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:47.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:47.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:47.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:47.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:47.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:47.681+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:48.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:47.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:48.186 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:47.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:48.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:47.922+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:48.187 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:47.923+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:48.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:48.196+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21803 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:48.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:48.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:48.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:48.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:48.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:48.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:48.903 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:48.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:48.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:48.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:48.904 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:48.662+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:49.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:48.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:49.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:48.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:48.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:48.914+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:49.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:49.326+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:49.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:49.326+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:49.672 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:49.326+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:50.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:50.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:50.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:49.682+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:50.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:49.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:50.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:49.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:50.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:49.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:50.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:49.866+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:50.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:50.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:50.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:50.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:50.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:50.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:51.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:50.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:51.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:50.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:51.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:50.654+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:50.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:50.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:50.824+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:50.824+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:51.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:51.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:51.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:51.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:51.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:52.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:51.656+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:51.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:51.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:51.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:51.811+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:52.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:52.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:52.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:52.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:52.694 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:52.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:53.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:52.704+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:53.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:52.704+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:53.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:52.704+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:52.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:52.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:52.789+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:53.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:52.789+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:53.475 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:53.200+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21880 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:53.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:53.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:53.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:53.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:53.476 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:53.359+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:53.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:53.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:53.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:53.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:53.770 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:53.691+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:54.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:53.780+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:54.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:53.780+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:54.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:53.780+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:54.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:53.780+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:54.663 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:54.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:54.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:54.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:54.664 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:54.319+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:54.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:54.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:55.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:54.673+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:54.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:54.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:54.752+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:54.752+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:55.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:55.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:55.300+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:56.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:55.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:56.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:55.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:56.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:55.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:55.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:55.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:55.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:55.725+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:56.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:56.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:56.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:56.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:56.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:56.291+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:56.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:56.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:56.735+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:57.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:56.735+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:57.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:56.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:57.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:56.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:57.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:56.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:57.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:57.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:57.266+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:57.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:57.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:57.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:57.714+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:58.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:57.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:58.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:57.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:58.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:57.686+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:58.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:40:58.203+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 21958 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:40:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:58.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:58.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:58.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:58.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:59.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:58.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:59.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:58.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:59.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:58.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:40:59.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:58.760+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:40:59.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:58.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:59.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:58.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:59.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:58.659+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:40:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:59.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:59.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:40:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:40:59.255+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:00.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:59.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:00.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:59.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:00.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:59.740+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:00.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:40:59.740+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:59.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:59.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:00.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:40:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:40:59.708+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:00.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:00.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:00.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:00.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:01.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:00.766+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:01.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:00.766+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:01.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:00.766+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:01.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:00.766+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:01.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:01.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:01.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:00.697+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:01.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:01.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:01.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:01.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:02.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:01.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:02.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:01.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:02.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:01.788+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:01.788+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:01.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:01.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:02.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:01.738+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:02.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:02.265+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:02.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:02.265+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:02.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:02.265+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:03.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:02.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:03.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:02.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:03.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:02.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:03.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:02.746+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:03.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:02.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:03.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:02.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:03.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:02.731+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:03.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:03.206+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22035 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:03.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:03.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:03.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:03.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:03.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:03.290+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:04.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:03.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:03.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:03.714+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:04.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:03.714+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:04.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:03.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:04.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:03.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:04.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:03.778+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:04.297+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:04.297+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:04.297+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:05.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:04.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:05.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:04.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:05.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:04.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:05.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:04.754+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:05.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:04.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:05.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:04.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:05.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:04.734+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:05.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:05.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:05.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:06.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:05.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:05.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:05.770+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:06.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:05.725+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:06.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:05.726+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:06.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:05.726+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:06.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:05.726+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:06.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:06.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:06.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:06.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:06.725 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:06.310+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:07.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:06.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:07.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:06.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:07.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:06.735+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:06.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:06.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:06.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:06.746+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:07.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:07.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:07.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:07.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:07.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:07.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:08.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:07.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:08.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:07.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:08.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:07.771+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:08.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:07.779+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:08.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:07.779+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:08.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:07.779+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:08.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:07.779+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:08.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:08.210+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22112 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:08.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:08.277+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:08.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:08.277+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:08.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:08.277+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:09.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:08.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:09.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:08.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:08.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:09.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:08.758+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:09.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:08.758+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:09.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:08.758+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:09.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:08.758+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:09.712 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:09.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:09.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:09.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:09.713 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:09.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:09.826+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:09.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:09.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:09.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:09.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:09.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:09.723+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:10.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:10.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:10.317+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:11.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:10.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:11.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:10.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:10.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:11.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:10.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:11.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:10.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:11.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:10.701+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:11.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:10.701+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:11.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:11.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:11.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:11.294+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:11.906+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:11.906+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:11.906+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:11.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:11.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:11.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:11.746+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:12.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:12.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:12.283+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:13.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:12.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:12.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:12.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:12.787+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:12.787+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:12.787+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:12.787+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:13.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:13.213+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22188 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:13.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:13.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:13.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:13.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:13.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:13.306+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:14.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:13.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:14.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:13.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:14.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:13.889+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:13.804+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:13.804+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:13.804+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:13.805+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:14.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:14.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:14.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:14.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:14.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:14.307+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:15.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:14.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:15.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:14.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:15.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:14.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:14.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:14.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:14.839+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:15.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:14.839+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:15.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:15.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:15.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:15.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:15.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:15.338+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:15.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:15.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:15.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:16.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:15.811+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:16.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:15.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:16.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:15.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:16.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:15.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:16.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:16.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:16.324+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:17.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:16.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:17.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:16.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:17.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:16.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:17.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:16.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:17.264 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:16.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:17.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:16.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:17.265 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:16.855+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:17.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:17.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:17.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:17.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:17.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:17.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:18.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:18.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:18.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:18.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:17.836+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:18.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:17.836+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:18.207 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:17.836+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:18.208 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:17.836+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:18.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:18.217+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22267 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:18.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:18.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:18.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:18.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:18.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:19.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:18.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:19.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:18.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:19.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:18.848+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:19.246 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:18.848+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:19.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:18.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:19.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:18.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:19.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:18.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:19.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:19.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:19.256+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:20.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:19.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:20.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:19.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:20.194 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:19.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:20.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:19.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:20.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:19.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:20.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:19.894+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:20.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:19.894+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:20.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:20.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:20.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:21.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:20.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:21.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:20.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:21.123 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:20.856+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:21.123 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:20.878+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:21.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:20.878+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:21.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:20.878+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:21.124 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:20.878+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:21.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:21.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:21.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:21.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:21.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:21.827+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:21.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:21.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:21.888+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:21.888+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:22.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:22.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:22.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:23.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:22.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:23.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:22.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:23.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:22.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:23.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:22.849+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:23.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:22.849+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:23.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:22.849+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:23.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:22.849+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:23.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:23.220+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22343 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:23.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:23.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:23.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:23.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:23.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:23.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:24.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:23.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:24.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:23.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:24.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:23.825+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:24.195 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:23.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:24.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:23.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:24.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:23.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:24.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:23.868+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:24.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:24.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:24.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:24.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:24.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:24.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:24.851+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:24.851+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:24.851+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:24.851+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:25.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:24.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:25.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:24.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:25.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:24.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:25.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:25.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:25.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:25.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:25.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:25.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:25.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:25.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:25.895+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:25.895+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:25.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:25.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:26.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:25.739+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:26.259+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:26.259+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:26.259+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:26.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:26.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:26.872+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:27.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:26.872+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:27.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:26.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:27.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:26.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:27.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:26.745+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:27.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:27.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:27.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:27.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:27.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:27.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:27.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:27.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:27.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:27.893+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:27.765+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:28.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:27.765+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:28.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:27.765+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:28.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:28.224+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22421 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:28.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:28.247+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:28.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:28.247+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:28.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:28.247+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:28.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:28.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:28.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:28.920+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:29.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:28.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:29.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:28.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:29.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:28.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:29.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:29.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:29.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:29.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:29.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:29.209+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:29.948 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:29.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:29.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:29.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:29.949 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:29.689+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:30.223 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:29.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:29.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:29.960+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:30.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:29.960+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:30.235+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:30.235+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:30.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:30.235+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:31.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:30.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:31.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:30.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:31.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:30.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:31.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:30.932+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:31.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:30.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:31.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:30.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:31.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:30.716+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:31.204+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:31.204+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:31.204+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:31.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:31.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:31.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:31.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:31.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:31.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:32.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:31.928+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:32.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:32.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:32.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:32.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:32.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:32.919+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:32.919+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:32.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:32.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:32.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:33.410 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:33.227+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22497 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:33.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:33.171+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:33.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:33.171+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:33.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:33.171+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:33.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:33.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:33.683+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:33.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:33.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:33.932+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:33.933+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:34.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:34.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:34.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:34.216+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:34.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:34.698+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:34.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:34.698+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:34.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:34.698+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:35.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:35.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:35.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:34.956+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:34.956+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:34.956+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:35.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:34.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:35.973 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:35.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:35.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:35.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:35.974 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:35.710+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:36.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:36.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:36.176+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:36.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:35.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:36.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:35.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:36.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:35.984+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:36.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:35.984+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:36.995 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:36.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:36.995 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:36.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:36.996 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:36.725+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:37.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:37.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:37.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:37.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:37.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:37.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:37.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:37.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:37.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:37.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:37.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:37.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:37.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:37.006+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:37.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:37.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:37.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:37.693+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:38.245 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:38.231+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22576 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:38.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:38.000+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:38.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:38.000+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:38.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:38.000+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:38.245 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:38.000+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:38.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:38.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:38.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:38.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:38.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:38.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:39.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:38.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:39.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:38.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:39.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:38.730+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:39.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:39.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:39.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:39.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:39.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:39.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:39.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.010+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:39.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:39.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:39.779+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:39.996+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:40.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:40.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:40.193+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:40.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:40.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:40.790+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:40.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:40.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:40.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:40.963+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:41.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:41.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:41.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:41.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:41.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:41.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:41.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:41.792+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:41.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:41.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:41.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:41.943+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:42.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:42.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:42.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:42.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:42.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:42.954 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:42.823+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:43.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:43.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:43.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:43.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:43.224 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:42.964+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:43.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:42.964+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:43.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:42.964+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:43.225 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:42.964+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:43.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:43.234+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22652 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:44.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:43.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:44.188 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:43.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:44.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:43.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:44.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:43.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:44.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:43.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:44.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:43.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:44.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:43.948+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:44.198+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:44.198+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:44.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:44.198+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:45.220 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:44.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:44.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:44.837+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:44.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:44.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:44.977+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:45.221 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:44.977+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:45.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:45.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:45.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:46.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:45.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:46.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:45.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:46.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:45.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:46.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:46.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:46.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:46.205 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:45.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:46.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:46.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:46.214+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:47.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:46.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:47.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:46.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:47.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:46.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:47.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:46.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:47.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:46.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:47.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:46.943+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:47.237 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:46.943+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:47.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:47.246+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:47.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:47.246+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:47.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:47.246+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:48.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:47.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:48.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:47.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:48.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:47.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:48.229 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:47.980+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:48.229 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:47.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:48.229 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:47.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:48.229 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:47.813+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:48.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:48.238+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22730 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:48.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:48.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:48.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:48.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:48.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:48.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:49.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:48.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:49.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:48.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:49.262 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:48.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:48.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:48.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:48.941+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:48.941+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:49.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:49.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:49.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:50.195 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:49.844+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:50.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:49.844+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:50.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:49.844+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:50.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:49.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:50.196 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:49.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:50.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:49.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:50.197 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:49.963+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:50.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:50.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:50.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:50.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:50.542 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:50.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:51.271 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:50.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:51.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:50.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:51.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:50.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:51.272 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:50.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:51.272 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:50.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:51.272 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:50.950+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:51.273 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:50.950+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:51.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:51.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:51.282+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:51.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:51.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:52.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:51.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:51.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:51.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:52.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:51.979+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:52.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:51.979+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:52.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:52.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:52.299+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:53.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:52.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:53.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:52.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:53.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:52.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:53.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:52.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:53.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:52.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:53.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:52.990+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:53.232 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:52.990+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:53.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:53.241+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22806 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:53.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:53.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:53.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:53.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:53.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:53.264+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:54.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:53.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:54.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:53.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:54.260 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:53.910+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:54.260 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:53.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:54.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:53.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:54.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:53.955+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:54.261 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:53.955+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:54.271+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:54.271+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:54.271+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:55.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:54.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:55.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:54.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:55.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:54.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:55.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:54.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:55.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:54.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:55.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:54.948+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:55.228 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:54.948+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:55.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:55.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:55.237+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:56.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:55.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:56.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:55.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:56.189 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:55.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:56.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:55.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:56.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:55.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:56.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:55.986+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:56.190 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:55.986+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:56.199+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:56.199+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:56.199+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:57.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:56.853+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:57.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:56.853+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:57.152 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:56.853+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:57.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:57.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:57.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:57.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:57.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:57.005+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:57.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:57.005+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:57.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:57.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:57.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:57.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:57.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:57.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:58.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:57.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:58.136 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:57.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:58.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:57.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:58.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:58.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:58.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:58.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:58.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:58.028+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:58.137 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:58.028+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:58.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:41:58.245+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22885 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:41:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:58.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:58.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:58.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:58.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:58.147+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:59.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:58.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:59.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:58.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:59.137 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:58.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:41:59.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:59.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:59.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:59.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:59.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:59.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:41:59.138 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:41:59.012+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:41:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:59.148+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:59.148+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:41:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:41:59.148+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:00.196 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:59.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:00.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:59.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:00.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:41:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:41:59.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:00.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:00.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:00.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:00.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:00.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:00.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:00.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:00.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:00.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:00.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:00.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:00.019+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:00.200 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:00.019+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:01.054+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:01.054+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:01.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:01.054+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:00.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:00.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:00.928+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:01.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:01.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:01.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:01.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:01.031+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:02.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:01.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:02.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:01.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:02.000 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:01.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:02.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:02.093+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:02.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:02.093+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:02.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:02.093+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:02.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:02.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:02.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:02.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:02.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:02.011+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:02.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:02.011+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:03.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:03.047+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:03.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:03.047+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:03.238 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:03.047+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:02.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:02.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:02.934+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:03.239 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.013+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:03.240 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.013+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:03.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:03.248+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 22961 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:04.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:04.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:04.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:04.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:04.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:04.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:03.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:03.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:04.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:03.950+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:04.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:04.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:04.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:03.972+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:04.966+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:04.966+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:04.966+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:04.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:04.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:04.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:04.946+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:05.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:05.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:05.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:05.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:05.084+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:06.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:06.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:06.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:06.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:06.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:06.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:05.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:05.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:05.998+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:05.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:05.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:06.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:05.946+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:06.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:05.946+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:07.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:06.987+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:07.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:06.987+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:07.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:06.987+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:07.146+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:07.146+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:07.146+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:06.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:06.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:06.935+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:07.192 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:06.935+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:08.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:08.120+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:08.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:08.120+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:08.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:08.120+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:08.242 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:07.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:08.242 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:07.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:08.243 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:07.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:08.243 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:07.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:08.243 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:07.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:08.243 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:07.971+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:08.243 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:07.971+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:08.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:08.252+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23039 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:08.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:08.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:09.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:08.985+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:08.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:08.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:08.924+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:09.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:08.924+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:09.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:09.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:09.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:09.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:09.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:09.106+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:10.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:09.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:10.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:09.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:10.197 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:09.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:10.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:10.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:10.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:09.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:09.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:10.198 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:09.934+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:10.199 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:09.934+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:11.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:10.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:11.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:10.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:11.172 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:10.988+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:11.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:10.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:11.172 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:10.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:11.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:10.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:11.173 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:10.886+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:11.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:11.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:11.182+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:11.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:11.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:11.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:11.867+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:11.867+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:11.867+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:12.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:11.867+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:12.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:12.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:12.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:13.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:12.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:13.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:12.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:13.181 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:12.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:13.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:12.842+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:13.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:12.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:13.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:12.843+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:13.182 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:12.843+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:13.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:13.255+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23115 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:13.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:13.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:13.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:13.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:13.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:13.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:14.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:13.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:14.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:13.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:14.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:13.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:13.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:13.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:14.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:13.799+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:14.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:13.799+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:14.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:14.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:14.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:15.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:14.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:15.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:14.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:15.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:14.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:14.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:14.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:14.764+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:15.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:14.764+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:15.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:15.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:15.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:15.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:15.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:15.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:16.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:16.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:16.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:15.932+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:16.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:15.784+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:16.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:15.784+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:16.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:15.784+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:16.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:15.784+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:16.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:16.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:16.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:17.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:16.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:17.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:16.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:17.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:16.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:17.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:16.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:17.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:16.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:17.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:16.795+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:17.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:16.795+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:17.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:17.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:17.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:18.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:17.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:18.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:17.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:18.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:17.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:18.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:17.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:18.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:17.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:18.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:17.827+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:18.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:17.827+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:18.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:18.258+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23190 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:18.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:18.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:18.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:19.139 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:18.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:18.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:18.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:18.810+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:18.810+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:18.810+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:19.140 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:18.810+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:19.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:19.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:19.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:19.151+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:20.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:19.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:20.176 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:19.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:20.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:19.967+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:19.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:19.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:19.826+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:20.177 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:19.826+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:20.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:20.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:20.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:20.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:21.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:20.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:21.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:20.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:21.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:20.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:21.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:20.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:21.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:20.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:21.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:20.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:21.142 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:20.829+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:21.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:21.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:21.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:21.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:21.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:22.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:21.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:21.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:21.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:21.871+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:22.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:21.871+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:22.140+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:22.140+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:22.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:22.140+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:23.115 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:22.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:22.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:22.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:22.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:22.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:22.831+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:23.116 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:22.831+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:23.375 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:23.262+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23266 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:23.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:23.127+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:23.375 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:23.127+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:23.376 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:23.127+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:24.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:23.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:24.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:23.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:24.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:23.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:23.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:23.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:23.793+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:24.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:23.793+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:24.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:24.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:24.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:24.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:24.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:24.087+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:24.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:24.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:24.840+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:25.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:25.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:25.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:24.805+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:24.805+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:24.805+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:25.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:24.805+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:26.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:25.793+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:26.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:25.793+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:26.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:25.793+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:25.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:26.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:25.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:26.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:25.791+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:26.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:25.791+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:26.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:26.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:26.091+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:26.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:26.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:26.811+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:27.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:27.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:27.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:27.057+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:26.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:26.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:26.832+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:27.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:26.833+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:28.255 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:28.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:28.255 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:28.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:28.256 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:28.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:28.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:27.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:28.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:27.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:28.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:27.853+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:28.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:27.853+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:28.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:27.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:28.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:27.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:28.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:27.854+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:28.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:28.265+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23344 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:29.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:28.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:29.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:28.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:29.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:28.816+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:29.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:29.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:29.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:29.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:29.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:29.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:28.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:29.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:28.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:29.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:28.844+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:29.293 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:28.844+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:30.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:29.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:30.069 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:29.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:30.070 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:29.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:30.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:29.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:30.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:29.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:30.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:29.811+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:30.070 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:29.811+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:30.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:30.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:30.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:31.054 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:30.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:30.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:30.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:30.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:30.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:30.802+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:31.055 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:30.802+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:31.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:31.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:31.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:31.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:32.016 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:31.865+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:31.865+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:31.865+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:31.815+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:31.815+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:31.815+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:32.017 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:31.816+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:32.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:32.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:32.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.027+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:33.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.994+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:33.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.994+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:33.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:32.994+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:32.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:32.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:32.870+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:32.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:32.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:33.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:32.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:33.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:32.775+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:33.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:33.269+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23420 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:33.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:33.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:34.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:33.890+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:33.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:33.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:33.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:33.741+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:33.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:33.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:34.041 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:33.983+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:35.021 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:34.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:34.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:34.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:34.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:34.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:34.754+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:35.022 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:34.754+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:35.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:35.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:35.031+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:36.008 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:35.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:36.009 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:35.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:36.009 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:35.907+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:36.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:35.704+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:36.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:35.704+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:36.009 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:35.704+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:36.010 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:35.704+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:36.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:36.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:36.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:36.019+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:37.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:36.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:36.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:37.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:36.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:36.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:37.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:36.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:37.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:36.723+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:37.043 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:36.723+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:37.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:37.059+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:37.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:37.059+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:37.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:37.059+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:38.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:37.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:38.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:37.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:38.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:37.769+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:38.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:37.769+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:38.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:37.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:38.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:37.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:38.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:37.923+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:38.384 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:38.273+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23499 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:38.384 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:38.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:38.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:38.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:38.385 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:38.098+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:39.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:38.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:39.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:38.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:39.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:38.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:39.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:38.781+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:39.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:38.781+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:39.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:38.781+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:39.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:38.781+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:39.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:39.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:39.095+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:39.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:39.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:40.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:39.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:40.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:39.738+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:39.738+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:39.738+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:40.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:39.738+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:40.092+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:40.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:40.092+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:40.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:40.092+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:40.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:40.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:40.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:40.774+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:40.774+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:40.774+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:40.774+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:41.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:41.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:41.089+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:41.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:41.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:41.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:41.775+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:41.776+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:41.776+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:42.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:41.776+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:42.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:42.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:42.102+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:42.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:42.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:42.948+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:42.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:42.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:43.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:42.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:43.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:42.760+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:43.390 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:43.276+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23575 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:43.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:43.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:43.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:43.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:43.391 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:43.079+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:44.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:43.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:44.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:43.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:44.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:43.935+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:44.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:43.785+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:44.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:43.785+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:44.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:43.785+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:44.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:43.785+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:44.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:44.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:44.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:45.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:44.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:45.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:44.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:45.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:44.954+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:45.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:44.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:45.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:44.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:45.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:44.771+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:45.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:44.771+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:45.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:45.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:45.136+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:46.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:45.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:46.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:45.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:46.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:45.981+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:46.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:45.737+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:46.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:45.737+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:46.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:45.737+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:46.042 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:45.737+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:46.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:46.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:46.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:47.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:46.745+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:46.745+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.006 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:46.745+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:46.745+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.016+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.016+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:47.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.016+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:47.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:47.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:47.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:47.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:47.292 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:47.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:47.718+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:47.718+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:47.718+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:47.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:47.718+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:48.269 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:48.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:48.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:48.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:48.270 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:48.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:48.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:48.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:48.270 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:47.991+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:48.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:48.280+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23653 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:48.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:48.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:48.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:48.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:48.972 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:48.670+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:49.242 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:48.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:49.242 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:48.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:49.242 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:48.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:49.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:49.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:49.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:49.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:49.647+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:49.991 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:49.647+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:49.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:49.647+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:49.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:49.647+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:50.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:50.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:50.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:50.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:50.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:50.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:50.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:50.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:50.240+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:50.959 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:50.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:50.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:50.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:50.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:50.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:50.960 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:50.627+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:51.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:51.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:51.215 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:50.970+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:51.226+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:51.226+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:51.226+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:51.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:51.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:51.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:51.971 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:51.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:52.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:52.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:52.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:52.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:52.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:52.213+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:52.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:51.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:52.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:51.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:52.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:51.982+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:52.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:52.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:52.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:52.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:52.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:52.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:52.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:52.690+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:53.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:52.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:53.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:52.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:53.246 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:52.941+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:53.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:53.284+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23729 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:53.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:53.257+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:53.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:53.257+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:53.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:53.257+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:53.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:53.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:53.944 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:53.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:53.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:53.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:53.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:53.695+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:54.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:53.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:54.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:53.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:54.282 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:53.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:54.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:54.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:54.293+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:54.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:54.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:54.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:55.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:54.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:54.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:54.730+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:54.730+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:55.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:55.304+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:55.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:55.304+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:55.708 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:55.304+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:56.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:55.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:56.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:55.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:56.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:55.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:55.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:55.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:55.719+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:56.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:55.719+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:56.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:56.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:56.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:57.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:56.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:57.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:56.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:57.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:56.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:56.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:57.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:56.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:57.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:56.760+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:57.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:56.760+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:57.704 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:57.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:57.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:57.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:57.705 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:57.311+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:58.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:57.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:58.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:57.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:58.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:57.855+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:58.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:57.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:58.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:57.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:58.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:57.716+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:58.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:57.716+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:58.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:42:58.288+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23808 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:42:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:58.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:58.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:58.287+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:59.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:58.833+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:59.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:58.833+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:59.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:58.833+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:42:59.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:58.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:59.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:58.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:59.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:58.702+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:42:59.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:58.702+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:42:59.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:59.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:59.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:42:59.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:42:59.245+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:00.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:59.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:00.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:59.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:00.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:42:59.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:00.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:59.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:00.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:59.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:00.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:59.689+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:00.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:42:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:42:59.689+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:00.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:00.211+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:00.211+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:00.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:00.211+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:00.913+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:00.913+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:01.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:00.913+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:01.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:00.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:00.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:00.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:01.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:00.690+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:01.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:01.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:01.248+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:01.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:01.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:01.876+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:02.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:01.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:01.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:01.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:01.720+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:02.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:02.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:02.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:02.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:02.499 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:02.252+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:02.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:02.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:02.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:02.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:02.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:02.721+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:03.264 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:02.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:03.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:02.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:03.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:02.904+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:03.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:03.291+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23884 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:03.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:03.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:03.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:03.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:03.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:03.276+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:04.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:03.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:04.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:03.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:04.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:03.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:04.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:03.698+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:03.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:03.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:04.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:03.898+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:04.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:04.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:04.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:04.230+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:05.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:04.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:05.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:04.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:05.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:04.741+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:05.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:04.741+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:05.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:04.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:05.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:04.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:05.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:04.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:05.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:05.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:05.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:05.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:06.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:05.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:06.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:05.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:06.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:05.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:06.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:05.699+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:05.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:05.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:06.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:05.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:06.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:06.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:06.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:06.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:06.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:06.228+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:07.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:06.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:07.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:06.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:07.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:06.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:06.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:06.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:06.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:06.663+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:07.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:07.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:07.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:07.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:07.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:07.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:07.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:07.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:07.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:07.639+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:08.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:07.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:08.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:07.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:08.141 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:07.869+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:08.409 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:08.295+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 23962 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:08.409 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:08.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:08.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:08.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:08.410 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:08.153+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:09.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:08.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:09.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:08.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:09.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:08.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:09.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:08.675+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:08.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:08.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:09.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:08.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:09.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:09.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:09.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:09.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:09.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:09.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:09.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:09.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:09.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:10.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:09.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:09.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:09.703+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:10.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:09.703+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:10.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:10.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:10.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:10.150+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:11.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:10.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:11.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:10.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:11.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:10.690+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:11.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:10.690+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:10.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:10.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:11.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:10.885+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:11.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:11.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:11.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:11.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:11.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:11.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:11.722+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:12.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:11.722+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:11.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:12.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:11.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:12.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:11.852+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:12.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:12.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:12.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:12.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:12.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:13.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:12.834+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:13.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:12.834+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:13.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:12.834+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:13.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:12.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:12.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:12.733+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:13.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:12.734+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:13.410 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:13.300+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24038 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:13.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:13.155+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:13.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:13.155+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:13.411 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:13.155+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:13.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:13.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:13.746+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:14.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:13.746+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:14.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:13.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:14.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:13.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:14.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:13.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:14.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:14.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:14.143+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:14.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:14.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:14.721+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:15.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:14.721+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:15.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:14.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:15.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:14.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:15.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:14.894+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:15.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:15.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:15.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:15.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:15.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:15.133+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:16.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:15.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:16.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:15.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:16.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:15.683+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:16.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:15.683+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:15.851+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:15.851+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:16.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:15.851+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:16.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:16.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:16.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:16.168+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:17.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:16.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:16.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:16.695+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:16.695+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:17.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:16.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:17.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:16.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:17.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:16.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:17.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:17.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:17.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:17.210+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:17.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:17.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:17.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:17.729+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:18.177 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:18.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:18.178 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:17.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:18.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:18.302+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24117 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:18.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:18.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:18.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:18.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:18.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:18.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:18.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:18.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:18.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:18.709+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:18.949 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:18.709+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:18.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:18.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:18.959+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:19.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:19.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:19.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:19.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:19.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:19.162+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:19.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:19.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:19.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:19.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:19.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:19.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:19.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:19.681+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:20.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:20.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:20.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:20.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:20.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:20.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:20.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:20.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:20.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:20.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:20.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:20.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:20.930 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:20.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:20.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:20.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:20.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:20.637+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:20.931 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:20.638+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:21.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:21.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:21.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:21.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:21.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:21.044+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:21.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:21.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:21.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:21.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:21.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:21.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:21.610+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:21.610+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:22.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:22.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:22.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:22.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:22.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:22.078+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:22.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:22.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:22.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:22.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:22.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:22.188+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:22.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:22.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:22.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:22.648+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:23.418 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:23.306+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24193 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:23.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:23.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:23.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:23.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:23.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:23.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:23.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:23.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:23.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:23.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:23.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:24.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:24.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:24.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:24.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:24.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:24.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:24.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:24.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:24.119+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:24.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:24.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:24.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:25.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:24.642+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:25.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:25.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:25.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:25.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:25.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:25.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:25.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:25.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:25.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:25.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:25.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:25.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:26.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:25.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:26.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:26.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:26.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:26.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:26.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:26.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:26.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:26.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:26.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:26.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:26.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:26.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:26.663+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:27.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:27.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:27.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:27.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:27.092+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:27.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:27.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:27.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:27.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:27.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:27.170+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:27.679+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:27.679+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:28.424 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:28.310+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24271 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:28.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:28.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:28.425 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:28.107+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:28.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:28.169+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:28.425 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:28.169+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:28.426 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:28.169+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:28.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:28.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:28.729+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:29.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:28.729+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:29.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:29.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:29.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:29.108+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:29.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:29.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:29.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:29.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:29.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:29.134+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:30.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:29.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:30.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:29.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:30.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:29.720+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:30.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:29.720+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:30.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:30.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:30.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:30.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:30.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:30.067+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:30.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:30.118+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:30.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:30.118+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:30.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:30.118+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:31.025 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:30.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:31.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:30.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:31.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:30.717+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:31.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:30.717+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:31.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:31.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:31.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:31.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:31.081+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:31.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:31.081+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:31.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:31.081+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:31.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:31.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:31.682+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:32.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:31.682+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:32.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:32.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:32.064+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:32.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:32.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:32.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:32.086+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:33.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:32.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:32.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.038 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:32.641+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:32.641+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:33.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:33.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:33.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:33.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:33.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:33.049+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:33.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:33.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:33.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:33.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:33.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:33.103+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:33.629 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:33.314+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24347 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:33.997 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:33.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:33.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:33.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:33.998 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:33.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:34.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:34.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:34.149+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:34.008+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:34.008+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:34.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:34.008+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:34.992 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:34.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:34.993 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:34.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:34.993 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:34.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:34.993 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:34.605+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.003+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:35.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:35.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:35.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:35.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:35.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:35.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:35.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:35.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:35.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:35.614+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:35.954 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:35.614+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:35.965+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:36.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:36.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:36.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:36.113+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:36.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:36.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:36.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:36.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:36.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:36.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:36.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:36.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:37.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:36.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:37.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:36.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:37.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:36.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:37.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:37.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:37.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:37.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:37.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:37.124+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:37.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:37.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:37.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:37.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:37.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:37.698+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:37.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:37.698+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:37.957+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:37.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:37.958+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:38.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:38.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:38.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:38.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:38.665 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:38.317+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24422 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:39.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:38.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:39.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:38.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:39.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:38.921+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:39.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:38.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:38.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:38.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:38.675+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:39.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:39.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:39.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:39.185+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:39.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:39.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:39.699+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:39.945 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:39.699+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:40.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:39.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:40.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:39.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:40.203 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:39.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:40.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:40.190+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:40.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:40.190+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:40.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:40.190+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:40.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:40.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:41.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:40.924+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:40.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:40.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:40.680+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:41.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:40.680+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:41.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:41.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:41.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:41.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:41.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:41.728+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:41.728+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:41.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:42.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:41.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:42.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:41.927+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:42.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:42.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:42.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:42.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:42.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:42.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:43.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:42.881+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:42.770+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:42.770+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:43.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:42.770+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:43.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:42.770+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:43.540 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:43.321+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24498 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:43.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:43.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:43.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:43.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:43.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:43.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:44.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:43.761+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:44.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:43.761+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:44.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:43.761+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:44.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:43.761+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:43.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:43.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:44.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:43.912+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:44.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:44.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:44.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:44.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:44.195+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:45.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:44.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:45.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:44.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:45.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:44.796+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:45.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:44.796+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:44.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:44.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:45.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:44.886+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:45.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:45.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:45.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:45.202+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:45.759+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:45.759+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:46.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:45.759+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:46.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:45.759+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:46.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:45.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:46.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:45.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:46.042 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:45.857+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:46.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:46.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:46.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:46.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:46.164+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:46.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:46.727+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:47.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:46.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:47.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:46.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:47.041 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:46.878+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:47.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:47.158+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:47.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:47.158+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:47.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:47.158+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:47.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:47.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:47.727+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:47.727+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:48.118 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:47.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:47.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:48.119 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:47.864+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:48.439 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:48.325+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24576 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:48.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:48.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:48.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:48.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:48.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:48.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:48.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:48.696+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:48.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:48.696+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:48.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:48.696+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:48.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:48.696+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:49.164 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:48.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:49.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:48.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:49.165 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:48.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:49.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:49.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:49.175+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:49.926 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:49.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:49.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:49.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:49.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:49.684+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:49.927 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:49.684+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:50.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:50.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:50.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:50.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:50.204 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:50.130+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:50.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:49.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:50.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:49.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:50.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:49.938+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:51.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:50.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:51.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:50.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:51.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:50.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:51.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:50.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:51.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:50.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:51.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:50.685+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:51.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:50.686+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:51.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:51.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:51.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:51.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:51.167+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:51.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:51.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:51.882+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:52.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:51.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:51.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:51.639+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:51.639+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:52.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:52.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:52.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:52.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:52.867 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:52.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.868 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:52.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.868 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:52.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:52.868 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:52.596+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:53.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:52.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:53.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:52.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:53.143 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:52.879+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:53.439 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:53.328+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24652 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:53.154+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:53.154+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:53.440 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:53.154+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:53.615+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:53.615+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:53.615+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:53.615+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:54.154 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:53.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:54.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:53.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:54.155 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:53.845+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:54.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:54.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:54.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:54.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:54.166+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:54.815+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:54.815+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:54.815+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:54.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:54.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:54.570+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:54.570+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:55.510 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:55.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:55.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:55.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:55.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:55.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:55.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:55.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:55.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:55.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:55.521+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:56.149 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:55.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:56.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:55.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:56.150 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:55.824+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:56.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:56.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:56.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:56.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:56.481 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:56.161+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:56.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:56.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:56.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:56.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:56.492+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:57.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:56.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:57.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:56.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:57.175 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:56.828+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:57.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:57.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:57.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:57.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:57.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:57.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:57.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:57.520+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:57.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:57.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:57.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:57.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:57.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:57.521+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:58.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:57.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:58.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:57.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:58.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:57.780+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:58.481 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:43:58.332+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24731 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:43:58.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:58.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:58.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:58.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:58.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:58.229+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:58.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:58.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:58.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:58.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:58.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:58.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:58.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:58.493+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:43:59.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:58.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:59.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:58.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:59.231 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:58.821+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:43:59.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:59.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:59.492 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:59.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:59.493 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:43:59.243+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:43:59.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:59.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:59.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:59.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:59.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:59.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:43:59.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:43:59.503+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:00.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:59.807+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:00.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:59.807+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:00.204 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:43:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:43:59.807+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:00.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:00.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:00.471 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:00.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:00.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:00.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:00.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:00.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:00.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:00.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:00.482+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:01.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:00.812+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:01.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:00.812+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:01.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:00.812+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:01.734 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:01.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:01.735 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:01.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:01.735 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:01.312+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:01.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:01.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:01.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:01.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:01.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:01.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:01.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:01.484+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:02.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:01.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:01.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:02.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:01.795+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:02.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:02.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:02.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:02.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:02.788 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:02.336+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:02.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:02.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:02.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:02.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:02.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:02.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:02.788 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:02.435+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:03.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:02.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:03.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:02.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:03.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:02.799+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:03.682 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:03.337+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24803 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:03.682 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:03.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:03.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:03.373+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:03.448+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:03.449+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:03.449+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:03.683 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:03.449+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:03.938 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:03.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:03.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:03.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:03.945 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:03.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:04.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:04.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:04.490+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:04.491+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:04.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:04.355+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:04.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:04.355+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:04.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:04.355+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:05.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:04.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:04.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:04.818+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:05.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:05.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:05.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:05.493+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:05.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:05.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:05.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:05.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:05.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:06.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:05.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:06.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:05.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:06.269 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:05.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:06.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:06.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:06.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:06.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:06.529 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:06.280+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:06.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:06.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:06.540+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:06.540+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:07.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:06.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:07.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:06.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:07.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:06.858+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:07.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:07.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:07.328+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:07.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:07.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:07.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:07.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:07.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:07.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:08.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:07.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:08.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:07.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:08.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:07.877+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:08.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:08.340+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24881 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:08.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:08.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:08.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:08.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:08.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:08.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:08.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:08.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:08.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:08.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:08.313+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:08.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:08.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:08.903+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:09.353+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:09.353+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:09.353+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:09.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:09.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:09.459+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:09.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:09.459+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:10.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:09.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:10.205 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:09.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:10.206 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:09.931+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:10.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:10.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:10.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:10.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:10.498 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:10.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:10.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:10.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:10.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:10.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:10.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:10.509+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:10.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:10.509+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:11.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:10.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:11.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:10.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:11.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:10.945+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:11.735 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:11.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:11.735 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:11.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:11.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:11.425+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:11.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:11.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:11.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:11.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:11.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:11.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:11.736 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:11.515+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:12.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:11.963+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:11.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:11.964+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:12.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:12.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:12.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:12.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:12.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:12.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:12.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:12.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:12.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:13.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:13.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:12.997+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:13.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:13.344+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 24957 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:13.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:13.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:13.500+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:13.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:13.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:13.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:13.572+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:13.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:13.572+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:14.257 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:14.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:14.258 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.006+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:14.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:14.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:14.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:14.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:14.983 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:14.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:14.983 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:14.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:14.983 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:14.564+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:14.984 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:14.564+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:15.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.994+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:15.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.994+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:15.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:14.994+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:15.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:15.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:15.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:15.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:15.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:15.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:15.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:15.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:15.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:15.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:15.599+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:15.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:15.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:15.973+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:16.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:16.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:16.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:16.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:16.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:16.583+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:16.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:16.583+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:16.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:16.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:16.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:16.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:17.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:16.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:17.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:16.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:17.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:16.962+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:17.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:17.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:17.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:17.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:17.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:17.472+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:17.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:17.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:17.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:17.541+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:17.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:17.541+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:18.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:17.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:18.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:17.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:18.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:17.936+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:18.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:18.347+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25036 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:18.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:18.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:18.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:18.500+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:18.500+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:18.500+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:18.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:18.500+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:19.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:18.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:18.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:18.942+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:19.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:19.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:19.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:19.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:19.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:19.482+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:19.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:19.482+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:19.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:19.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:19.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:19.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:19.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:19.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:20.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:19.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:20.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:19.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:20.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:19.971+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:20.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:20.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:20.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:20.510+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:20.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:20.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:20.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:20.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:20.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:20.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:20.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:20.979+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:21.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:21.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:21.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:21.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:21.737 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:21.494+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:21.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:21.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:21.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:21.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:21.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:21.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:21.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:21.468+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:22.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:22.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:22.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:22.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:22.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:22.024+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:22.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:22.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:22.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:22.465+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:22.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:22.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:22.515+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:22.515+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:23.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:23.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:23.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:23.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:23.002+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:23.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:23.351+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25112 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:23.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:23.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:23.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:23.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:23.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:23.463+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:23.463+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:23.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:23.463+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:24.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:24.018+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:24.018+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:24.018+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:24.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:24.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:24.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:24.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:24.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:24.423+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:24.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:24.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:24.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:24.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:24.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:24.471+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:25.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:25.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:25.025+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:25.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:25.392+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:25.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:25.392+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:25.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:25.392+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:25.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:25.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:25.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:25.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:25.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:25.481+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:25.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:25.481+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:26.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:26.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:26.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.011+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:26.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:26.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:26.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:26.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:26.387+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:26.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:26.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:26.462+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:26.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:26.462+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.999+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.999+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:27.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:26.999+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:27.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:27.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:27.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:27.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:27.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:27.385+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:27.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:27.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:27.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:27.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:27.468+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:28.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:28.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:28.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:27.951+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:28.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:28.355+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25190 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:28.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:28.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:28.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:28.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:28.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:28.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:28.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:28.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:28.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:28.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:28.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:29.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:28.944+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:29.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:28.944+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:29.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:28.944+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:29.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:29.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:29.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:29.457+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:29.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:29.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:29.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:29.434+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:30.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:29.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:30.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:29.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:30.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:29.955+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:30.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:30.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:30.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:30.465+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:30.465+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:30.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:30.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:30.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:30.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:30.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:30.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:31.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:30.972+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:31.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:31.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:31.736 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:31.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:31.737 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:31.397+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:31.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:31.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:31.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:31.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:31.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:31.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:31.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:31.436+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:32.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:31.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:31.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:32.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:31.978+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:32.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:32.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:32.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:32.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:32.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:32.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:32.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:32.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:32.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:32.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:32.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:32.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:32.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:33.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:32.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:33.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:32.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:33.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:32.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:33.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:33.358+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25267 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:33.400+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:33.400+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:33.400+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:33.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:33.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:33.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:33.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:33.483+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:34.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:33.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:34.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:33.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:34.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:33.892+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:34.658 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:34.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:34.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:34.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:34.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:34.506+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:34.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:34.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:34.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:34.403+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:34.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:34.403+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:34.662 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:34.403+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:35.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:34.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:35.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:34.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:35.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:34.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:35.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:35.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:35.394+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:35.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:35.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:35.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:35.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:35.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:35.484+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:36.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:35.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:35.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:36.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:35.849+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:36.461+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:36.461+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:36.461+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:36.462+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:36.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:36.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:36.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:36.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:36.405+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:37.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:36.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:37.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:36.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:37.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:36.863+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:37.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:37.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:37.407+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:37.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:37.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:37.422+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:37.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:37.422+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:38.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:37.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:38.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:37.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:38.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:37.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:38.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:38.362+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25345 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:38.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:38.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:38.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:38.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:38.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:38.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:38.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:38.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:38.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:38.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:38.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:38.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:38.426+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:39.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:38.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:39.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:38.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:39.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:38.861+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:39.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:39.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:39.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:39.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:39.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:39.384+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:39.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:39.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:39.354+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:39.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:39.354+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:40.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:39.873+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:40.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:39.873+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:40.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:39.873+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:40.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:40.401+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:40.401+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:40.401+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:40.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:40.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:40.307+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:40.307+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:41.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:40.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:41.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:40.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:41.276 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:40.843+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:41.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:41.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:41.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:41.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:41.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:41.287+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:41.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:41.287+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:41.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:41.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:41.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:41.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:41.541 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:41.432+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:42.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:41.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:41.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:42.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:41.831+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:42.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:42.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:42.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:42.439+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:42.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:42.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:42.306+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:42.306+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:43.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:42.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:43.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:42.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:43.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:42.830+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:43.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:43.366+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25421 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:43.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:43.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:43.459+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:43.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:43.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:43.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:43.305+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:43.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:43.305+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:44.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:43.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:44.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:43.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:44.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:43.874+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:44.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:44.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:44.345+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:44.345+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:44.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:44.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:44.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:44.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:44.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:44.417+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:45.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:44.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:45.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:44.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:45.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:44.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:45.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:45.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:45.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:45.446+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:45.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:45.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:45.357+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:45.357+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:46.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:45.871+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:46.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:45.871+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:46.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:45.871+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:46.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:46.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:46.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:46.429+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:46.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:46.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:46.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:46.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:46.343+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:47.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:46.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:46.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:46.908+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:47.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:47.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:47.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:47.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:47.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:47.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:47.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:47.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:47.436+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:48.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:47.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:48.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:47.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:48.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:47.893+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:48.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:48.370+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25499 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:48.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:48.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:48.462+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:48.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:48.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:48.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:48.411+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:48.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:48.411+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:49.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:48.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:49.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:48.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:49.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:48.883+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:49.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:49.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:49.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:49.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:49.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:49.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:49.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:49.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:49.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:49.486+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:50.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:49.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:50.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:49.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:50.208 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:49.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:50.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:50.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:50.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:50.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:50.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:50.466+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:50.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:50.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:50.368+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:50.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:50.368+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:51.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:50.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:51.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:50.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:51.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:50.896+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:51.738 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:51.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:51.738 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:51.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:51.739 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:51.455+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:51.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:51.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:51.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:51.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:51.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:51.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:51.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:51.370+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:52.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:51.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:51.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:52.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:51.940+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:52.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:52.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:52.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:52.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:52.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:52.460+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:52.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:52.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:52.370+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:52.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:52.370+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:53.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:53.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:53.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:52.916+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:53.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:53.374+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25576 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:53.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:53.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:53.421+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:53.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:53.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:53.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:53.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:53.335+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:54.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:53.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:54.292 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:53.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:54.293 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:53.922+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:54.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:54.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:54.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:54.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:54.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:54.458+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:54.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:54.312+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:54.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:54.312+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:54.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:54.312+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:54.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:54.312+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:55.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:54.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:55.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:54.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:55.278 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:54.918+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:55.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:55.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:55.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:55.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:55.410+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:55.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:55.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:55.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:55.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:55.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:55.289+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:56.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:55.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:56.234 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:55.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:56.235 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:55.956+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:56.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:56.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:56.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:56.438+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:56.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:56.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:56.245+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:56.245+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:57.265 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:56.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:57.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:56.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:57.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:56.926+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:57.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:57.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:57.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:57.480+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:57.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:57.276+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:57.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:57.276+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:57.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:57.277+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:57.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:57.277+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:58.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:57.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:58.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:57.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:58.232 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:57.897+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:58.490 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:44:58.377+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25653 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:44:58.490 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:58.242+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:58.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:58.242+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:58.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:58.242+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:58.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:58.243+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:58.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:58.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:58.516+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:59.239 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:58.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:59.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:58.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:59.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:58.888+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:44:59.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:59.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:59.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:59.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:59.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:59.250+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:44:59.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:44:59.250+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:44:59.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:59.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:59.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:59.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:44:59.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:44:59.513+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:00.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:59.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:00.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:59.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:00.210 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:44:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:44:59.887+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:00.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:00.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:00.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:00.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:00.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:00.289+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:00.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:00.289+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:00.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:00.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:00.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:00.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:00.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:00.541+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:01.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:00.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:01.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:00.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:01.266 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:00.933+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:01.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:01.277+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:01.277+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:01.277+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:01.277+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:01.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:01.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:01.904 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:01.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:01.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:01.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:02.274 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:01.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:02.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:01.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:02.275 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:01.915+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:02.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:02.286+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:02.286+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:02.286+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:02.286+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:02.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:02.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:02.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:02.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:02.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:02.572+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:03.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:02.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:03.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:02.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:03.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:02.901+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:03.496 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:03.381+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25730 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:03.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:03.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:03.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:03.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:03.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:03.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:03.497 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:03.255+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:03.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:03.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:03.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:03.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:03.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:03.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:04.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:03.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:04.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:03.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:04.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:03.939+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:04.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:04.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:04.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:04.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:04.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:04.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:04.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:04.210+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:04.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:04.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:04.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:04.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:04.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:04.532+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:05.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:04.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:05.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:04.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:05.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:04.900+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:05.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:05.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:05.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:05.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:05.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:05.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:05.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:05.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:05.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:05.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:05.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:05.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:05.506+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:06.252 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:05.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:06.252 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:05.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:06.253 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:05.895+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:06.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:06.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:06.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:06.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:06.509 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:06.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:06.510 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:06.263+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:06.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:06.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:06.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:06.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:06.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:06.520+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:07.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:06.850+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:07.244 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:06.850+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:07.245 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:06.850+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:07.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:07.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:07.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:07.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:07.255+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:07.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:07.255+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:07.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:07.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:07.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:07.542+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:08.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:07.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:08.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:07.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:08.207 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:07.804+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:08.501 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:08.385+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25807 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:08.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:08.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:08.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:08.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:08.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:08.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:08.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:08.218+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:08.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:08.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:08.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:08.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:08.752 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:08.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:09.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:08.763+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:09.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:08.763+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:09.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:08.763+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:09.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:09.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:09.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:09.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:09.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:09.218+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:09.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:09.218+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:10.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:09.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:10.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:09.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:10.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:09.618+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:09.761+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:09.761+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:10.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:09.761+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:10.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:10.205+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:10.206+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:10.206+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:10.206+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:11.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:10.617+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:11.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:10.617+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:11.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:10.617+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:11.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:10.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:11.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:10.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:11.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:10.781+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:11.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:11.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:11.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:11.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:11.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:11.223+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:11.522 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:11.223+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:11.786 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:11.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:11.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:11.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:11.787 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:11.631+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:12.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:11.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:11.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:12.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:11.797+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:12.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:12.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:12.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:12.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:12.204+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:13.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:12.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:13.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:12.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:13.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:12.829+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:13.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:12.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:13.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:12.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:13.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:12.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:36.146635+0000 (oldest deadline 2023-12-19T19:34:56.146635+0000) 2023-12-19T19:45:13.503 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:13.389+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25885 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:13.503 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:13.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:13.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:13.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:13.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:13.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:13.504 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:13.184+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:14.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:13.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:14.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:13.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:14.219 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:13.872+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:14.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:14.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:14.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:14.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:14.230+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:15.179 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:14.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:15.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:14.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:15.180 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:14.884+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:38.485301+0000 (oldest deadline 2023-12-19T19:34:58.485301+0000) 2023-12-19T19:45:15.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:15.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:15.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:15.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:15.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:15.190+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:15.573 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:15.191+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:16.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:16.167+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:16.167+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:16.167+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:34:41.600388+0000 (oldest deadline 2023-12-19T19:35:01.600388+0000) 2023-12-19T19:45:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:16.167+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:17.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:17.156+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:18.507 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:18.393+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 25962 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:18.508 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:18.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:19.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:19.150+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:20.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:20.107+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:21.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:21.134+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:22.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:22.159+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:23.513 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:23.397+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26039 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:23.513 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:23.179+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:24.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:24.198+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:25.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:25.214+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:26.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:26.244+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:27.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:27.204+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:28.515 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:28.401+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26116 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:28.515 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:28.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:29.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:29.187+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:30.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:30.169+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:31.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:31.129+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:32.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:32.106+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:33.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:33.060+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:33.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:33.405+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26194 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:34.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:34.106+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:35.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:35.144+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:35.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:36.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:35.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:36.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:35.551+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:36.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:36.145+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:37.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:36.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:37.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:36.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:37.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:36.555+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:37.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:37.118+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:37.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:37.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:37.087+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:37.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:37.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:37.793 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:37.599+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:38.397 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:38.102+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:38.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:38.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:38.409+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26271 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:38.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:38.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:38.640+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:39.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:39.071+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:39.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:39.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:39.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:39.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:39.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:39.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:40.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:39.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:39.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:40.026 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:39.680+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:40.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:40.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:40.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:40.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:40.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:40.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:40.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:40.036+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:40.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:40.036+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:40.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:40.036+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:40.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:40.036+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:41.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:40.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:41.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:40.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:41.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:40.715+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:41.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:41.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:41.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:41.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:41.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:41.110+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:41.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:41.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:41.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:41.051+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:41.292 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:41.051+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:41.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:42.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:41.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:42.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:41.683+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:42.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:42.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:42.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:42.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:42.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:42.126+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:42.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:42.076+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:42.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:42.077+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:42.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:42.077+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:42.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:42.077+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:43.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:43.401 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:43.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:43.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:43.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:43.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:43.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:43.103+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:43.402 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:43.103+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:43.692 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:43.412+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26348 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:44.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:43.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:44.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:43.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:44.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:43.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:44.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:44.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:44.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:44.061+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:44.061+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:44.061+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:44.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:44.062+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:45.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:44.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:45.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:44.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:45.034 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:44.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:45.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:45.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:45.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:45.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:45.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:45.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:45.045+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:45.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:45.045+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:46.027 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:45.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:46.028 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:45.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:46.028 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:45.710+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:46.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:46.170+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:46.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:46.170+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:46.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:46.170+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:46.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:46.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:46.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:46.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:46.039+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:46.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:46.039+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:47.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:46.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:47.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:46.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:47.004 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:46.676+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:47.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:47.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:47.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:47.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:47.015+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:47.015+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:47.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:47.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:47.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:47.208+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:47.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:47.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:47.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:47.647+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:48.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:48.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:48.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:48.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:48.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:48.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:48.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:48.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:48.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:48.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:48.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:48.050+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:48.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:48.050+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:48.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:48.416+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26425 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:48.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:48.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:48.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:48.649+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:49.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:49.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:49.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:49.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:49.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:49.185+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:49.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:49.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:49.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:49.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:49.012+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:49.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:49.012+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:50.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:49.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:50.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:49.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:50.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:49.620+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:50.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:50.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:50.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:50.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:50.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:50.054+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:50.055+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:50.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:50.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:50.644+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:51.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:51.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:51.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:51.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:51.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:51.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:51.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:51.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:51.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:51.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:51.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:51.071+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:51.741 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:51.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:51.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:51.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:51.742 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:51.597+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:52.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:52.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:52.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:52.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:52.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:52.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:52.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:52.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:52.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:52.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:52.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:52.024+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:52.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:52.024+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:53.037 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:52.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:53.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:52.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:53.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:52.636+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:53.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:53.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:53.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:53.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:53.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:53.049+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:53.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:53.049+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:53.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:53.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:53.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:53.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:53.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:53.132+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:53.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:53.420+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26503 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:53.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:53.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:53.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:53.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:54.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:54.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:54.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:54.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:54.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:54.175+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:54.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:54.075+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:54.530 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:54.075+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:54.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:54.075+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:54.531 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:54.075+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:54.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:54.594+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:54.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:54.594+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:54.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:54.594+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:55.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:55.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:55.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:55.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:55.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:55.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:55.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:55.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:55.042+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:55.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:55.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:56.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:55.584+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:56.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:56.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:56.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:56.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:56.087+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:56.087+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:56.087+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:56.087+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:56.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:56.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:57.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:56.616+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:57.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:57.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:57.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:57.058+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:57.059+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:57.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:57.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:57.615+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:58.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:58.151+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:58.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:58.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:58.025+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:58.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:58.025+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:45:58.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:45:58.424+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26580 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:45:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:58.655+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:58.655+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:58.655+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:45:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:59.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:59.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:45:59.171+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:59.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:59.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:59.038+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:45:59.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:45:59.038+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:59.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:59.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:00.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:45:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:45:59.678+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:00.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:00.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:00.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:00.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:00.085+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:00.085+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:00.085+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:00.085+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:00.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:00.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:00.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:01.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:01.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:01.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:01.135+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:01.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:01.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:01.071+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:01.071+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:01.692+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:02.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:01.692+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:02.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:01.692+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:02.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:02.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:02.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:02.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:02.059+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:02.059+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:02.059+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:02.059+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:02.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:02.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:02.688+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:03.417 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:03.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:03.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:03.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:03.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:03.108+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:03.418 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:03.108+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:03.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:03.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:03.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:03.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:03.419 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:03.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:03.675 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:03.428+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26657 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:04.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:03.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:04.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:03.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:04.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:03.686+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:04.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:04.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:04.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:04.156+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:04.093+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:04.093+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:04.093+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:04.093+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:05.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:04.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:05.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:04.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:05.033 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:04.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:05.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:05.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:05.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:05.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:05.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:05.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:05.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:05.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:05.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:05.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:05.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:05.044+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:05.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:05.044+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:05.990 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:05.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:05.990 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:05.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:05.990 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:05.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:06.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:06.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:06.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:06.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:06.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:06.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:06.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:06.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:06.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:06.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:06.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:06.001+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:06.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:06.001+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:07.010 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:06.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:07.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:06.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:07.011 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:06.632+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:07.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:07.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:07.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:07.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:07.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:07.104+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:07.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:07.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:07.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:07.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:07.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:07.022+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:07.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:07.022+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:07.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:07.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:07.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:07.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:07.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:07.666+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:08.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:08.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:08.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:08.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:08.042+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:08.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:08.043+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:08.790 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:08.433+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26734 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:08.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:08.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:08.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:08.651+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:09.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:09.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:09.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:09.100+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.006+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:09.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.006+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:09.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:09.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:09.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:09.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:09.962 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:09.677+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:10.213 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:10.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:10.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:10.052+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.973+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:10.214 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:09.973+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:10.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:10.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:10.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:10.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:10.940 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:10.671+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:11.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:11.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:11.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:11.036+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:10.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:10.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:10.951+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:11.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:10.951+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:11.942 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:11.623+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:11.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:11.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:11.943 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:11.624+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:12.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:12.079+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:12.079+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:12.079+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:11.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:11.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:11.954+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:12.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:11.954+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:12.905 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:12.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:12.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:12.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:12.906 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:12.585+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:13.039+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:13.039+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:13.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:13.039+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:13.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:12.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:13.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:12.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:13.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:12.917+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:13.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:12.917+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:13.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:13.437+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26812 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:13.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:13.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:13.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:13.791 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:13.590+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:14.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:14.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:14.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:14.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:14.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:14.068+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:14.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:13.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:14.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:13.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:13.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:14.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:13.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:14.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:14.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:14.976 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:14.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:14.977 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:14.628+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:15.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:15.042+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:15.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:15.042+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:15.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:15.042+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:14.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:14.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:14.988+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:15.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:14.988+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:15.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:15.589+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:15.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:15.589+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:15.956 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:15.589+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:16.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:16.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:16.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:16.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:16.053+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:15.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:15.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:15.967+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:16.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:15.967+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:16.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:16.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:16.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:16.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:16.546+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:17.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:17.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:17.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:17.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:17.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:17.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:17.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:16.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:17.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:16.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:17.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:16.997+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:17.291 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:16.997+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:17.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:17.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:17.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:17.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:17.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:17.574+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:18.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:18.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:18.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:18.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:18.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:18.090+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:18.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:17.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:18.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:17.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:18.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:17.963+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:18.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:17.963+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:18.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:18.441+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26889 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:18.591+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:18.591+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:18.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:18.591+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:19.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:19.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:19.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:19.102+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:18.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:18.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:18.914+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:19.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:18.914+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:20.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:19.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:20.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:19.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:20.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:19.626+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:20.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:19.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:20.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:19.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:20.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:19.873+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:20.041 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:19.873+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:20.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:20.134+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:20.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:20.134+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:20.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:20.134+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:21.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:20.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:21.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:20.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:21.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:20.629+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:21.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:20.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:21.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:20.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:21.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:20.868+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:21.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:20.868+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:21.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:21.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:21.088+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:21.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:21.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:21.660+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:21.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:21.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:21.886+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:22.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:21.886+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:22.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:22.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:22.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:22.073+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:23.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:22.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:23.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:22.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:23.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:22.696+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:22.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:22.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:22.865+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:23.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:22.865+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:23.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:23.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:23.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:23.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:23.434 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:23.106+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:23.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:23.444+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 26966 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:23.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:23.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:23.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:23.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:23.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:23.657+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:24.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:23.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:24.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:23.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:24.114 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:23.840+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:24.115 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:23.840+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:24.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:24.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:24.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:24.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:24.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:24.125+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:25.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:24.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:25.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:24.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:24.829+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:25.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:24.829+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:24.609+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:24.609+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:25.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:24.609+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:25.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:25.158+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:25.158+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:25.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:25.158+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:25.843 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:25.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:25.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:25.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:25.844 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:25.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:26.171 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:25.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:26.171 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:25.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:26.171 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:25.855+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:26.171 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:25.855+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:26.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:26.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:26.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:26.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:26.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:26.183+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:26.868 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:26.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:26.868 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:26.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:26.868 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:26.625+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:27.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:26.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:27.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:26.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:27.152 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:26.880+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:27.153 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:26.880+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:27.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:27.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:27.164+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:27.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:27.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:27.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:27.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:27.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:27.633+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:28.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:27.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:28.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:27.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:28.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:27.882+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:28.181 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:27.882+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:28.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:28.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:28.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:28.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:28.438 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:28.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:28.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:28.449+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27043 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:28.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:28.604+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:28.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:28.604+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:28.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:28.604+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:29.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:28.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:29.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:28.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:29.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:28.893+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:29.193 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:28.893+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:29.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:29.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:29.205+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:29.871 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:29.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:29.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:29.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:29.872 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:29.612+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:30.169 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:29.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:30.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:29.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:30.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:29.883+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:30.170 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:29.883+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:30.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:30.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:30.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:30.891 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:30.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:30.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:30.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:30.892 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:30.586+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:31.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:30.903+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:31.155 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:30.903+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:31.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:30.903+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:31.156 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:30.903+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:31.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:31.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:31.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:31.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:31.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:31.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:32.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:31.630+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:31.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:31.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:31.859+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:32.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:31.859+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:32.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:32.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:32.130+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:33.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:32.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:33.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:32.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:33.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:32.582+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:32.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:32.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:32.821+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:33.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:32.821+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:33.441 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:33.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:33.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:33.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:33.442 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:33.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:33.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:33.453+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27121 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:33.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:33.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:33.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:33.601+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:34.164 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:33.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:34.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:33.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:34.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:33.866+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:34.165 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:33.867+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:34.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:34.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:34.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:34.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:34.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:34.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:34.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:34.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:34.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:34.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:34.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:34.650+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:35.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:34.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:34.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:34.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:34.909+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:35.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:35.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:35.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:35.146+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:35.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:35.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:35.681+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:36.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:35.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:36.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:35.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:36.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:35.920+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:36.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:35.920+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:36.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:36.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:36.177+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:36.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:36.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:36.712+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:37.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:36.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:37.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:36.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:37.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:36.909+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:37.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:36.909+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:37.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:37.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:37.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:37.220+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:37.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:37.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:37.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:37.663+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:38.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:37.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:38.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:37.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:38.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:37.899+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:38.189 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:37.899+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:38.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:38.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:38.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:38.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:38.446 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:38.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:38.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:38.457+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27198 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:38.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:38.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:38.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:38.700+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:39.230 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:38.911+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:39.231 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:38.911+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:39.231 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:38.911+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:39.231 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:38.911+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:39.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:39.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:39.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:39.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:39.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:39.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:40.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:39.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:40.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:39.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:40.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:39.697+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:40.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:39.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:40.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:39.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:40.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:39.906+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:40.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:39.906+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:40.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:40.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:40.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:40.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:40.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:40.228+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:41.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:40.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:40.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:40.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:40.862+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:40.862+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:40.862+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:41.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:40.862+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:41.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:41.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:41.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:41.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:41.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:41.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:41.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:41.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:42.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:41.704+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:41.828+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:41.828+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:41.828+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:42.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:41.828+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:42.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:42.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:42.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:42.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:42.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:42.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:42.672+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:42.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:42.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:42.874+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:43.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:42.874+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:43.539 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:43.461+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27275 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:43.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:43.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:43.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:43.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:43.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:43.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:43.889 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:43.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:43.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:43.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:43.890 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:43.714+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:44.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:43.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:44.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:43.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:44.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:43.900+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:44.238 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:43.900+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:44.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:44.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:44.249+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:44.917 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:44.669+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:44.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:44.669+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:44.918 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:44.669+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:45.262 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:44.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:45.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:44.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:45.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:44.928+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:45.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:44.929+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:45.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:45.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:45.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:45.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:45.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:45.716+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:45.957+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:45.957+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:46.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:46.252+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:46.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:46.252+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:46.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:46.252+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:46.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:46.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:46.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:46.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:46.985 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:46.689+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:47.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:47.237+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:47.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:47.237+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:47.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:47.237+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:46.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:46.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:46.996+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:47.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:46.996+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:47.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:47.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:47.721+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:48.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:48.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:48.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:48.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:48.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:48.010+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:48.251 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:48.010+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:48.539 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:48.465+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27351 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:48.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:48.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:48.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:48.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:48.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:48.262+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:48.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:48.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:48.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:48.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:48.992 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:48.726+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:49.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:49.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:49.003+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:49.263 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:49.003+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:49.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:49.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:49.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:49.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:49.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:49.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:50.020 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:49.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:50.020 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:49.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:50.020 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:49.703+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:50.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:50.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:50.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:50.235+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:50.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:50.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:50.031+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:50.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:50.031+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:51.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:50.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:51.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:50.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:51.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:50.687+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:51.275+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:51.275+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:51.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:51.275+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:51.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:51.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:51.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:51.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:51.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:51.081+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:51.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:51.081+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:51.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:51.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:51.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:52.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:52.311+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:52.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:52.311+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:52.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:52.311+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:52.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:52.117+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:52.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:52.117+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:52.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:52.117+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:52.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:52.117+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:52.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:52.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:52.690+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:53.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:53.307+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:53.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:53.307+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:53.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:53.307+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:53.458 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:53.125+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:53.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:53.125+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:53.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:53.125+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:53.459 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:53.125+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:53.708 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:53.469+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27429 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:53.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:53.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:53.709 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:53.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:53.714 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:53.719+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:54.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:54.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:54.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:54.327+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:54.122+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:54.122+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:54.122+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:54.122+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:55.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:54.694+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:55.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:54.694+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:55.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:54.694+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:55.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:55.119+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:55.119+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:55.119+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:55.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:55.119+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:56.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:55.740+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:56.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:56.296+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:56.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:56.296+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:56.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:56.296+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:56.121+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:56.121+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:56.121+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:56.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:56.121+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:56.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:57.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:56.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:57.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:56.780+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:57.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:57.301+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:57.301+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:57.301+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:57.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:57.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:57.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:57.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:57.127+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:57.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:57.127+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:57.742+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:57.742+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:57.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:57.742+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:58.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:58.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:58.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:58.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:58.462 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:58.351+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:58.462 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:58.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:58.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:58.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:58.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:58.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:58.463 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:58.169+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:46:58.730 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:46:58.473+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27506 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:46:59.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:58.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:59.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:58.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:59.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:58.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:46:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:59.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:59.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:46:59.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:46:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:59.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:59.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:59.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:46:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:46:59.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:59.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:59.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:46:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:46:59.769+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:00.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:00.329+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:00.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:00.329+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:00.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:00.329+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:00.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:00.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:00.169+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:00.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:00.169+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:00.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:00.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:00.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:01.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:01.344+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:01.344+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:01.344+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:01.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:01.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:01.172+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:01.172+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:01.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:01.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:01.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:02.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:02.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:02.326+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:02.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:02.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:02.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:02.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:03.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:02.854+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:03.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:02.854+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:03.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:02.854+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:03.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:03.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:03.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:03.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:03.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:03.187+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:03.466 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:03.187+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:03.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:03.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:03.467 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:03.308+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:03.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:03.477+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27583 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:03.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:03.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:04.221 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:03.884+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:04.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:04.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:04.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:04.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:04.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:04.232+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:04.232+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:05.245 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:04.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:05.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:04.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:05.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:04.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:05.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:05.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:05.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:05.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:05.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:05.274+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:05.274+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:06.212 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:05.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:06.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:05.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:06.213 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:05.866+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:06.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:06.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:06.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:06.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:06.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:06.229+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:06.229+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:07.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:06.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:07.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:06.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:07.189 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:06.886+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:07.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:07.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:07.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:07.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:07.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:07.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:07.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:07.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:07.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:07.261+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:07.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:07.261+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:08.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:07.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:08.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:07.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:08.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:07.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:08.227 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:08.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:08.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:08.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:08.228 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:08.181+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:08.481 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:08.481+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27660 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:08.481 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:08.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:08.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:08.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:08.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:08.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:08.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:08.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:09.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:09.155+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:09.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:09.155+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:09.236 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:09.155+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:09.237 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:08.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:09.237 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:08.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:09.237 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:08.970+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:09.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:09.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:09.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:09.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:09.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:09.247+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:09.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:09.247+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:10.198 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:10.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:10.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:10.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:10.199 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:10.179+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:10.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:09.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:10.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:09.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:10.199 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:09.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:10.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:10.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:10.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:10.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:10.210+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:10.210+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:11.192 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:10.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:11.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:10.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:11.193 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:10.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:11.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:11.204+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:11.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:11.204+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:11.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:11.204+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:11.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:11.260+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:11.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:11.260+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:11.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:11.260+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:11.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:11.260+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:11.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:11.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:12.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:11.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:12.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:12.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:12.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:12.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:12.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:12.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:12.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:12.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:12.230+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:12.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:12.230+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:13.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:12.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:13.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:12.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:13.177 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:12.857+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:13.188+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:13.188+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:13.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:13.188+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:13.474 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:13.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:13.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:13.199+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:13.475 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:13.199+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:13.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:13.485+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27735 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:14.166 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:13.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:14.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:13.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:14.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:13.905+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:14.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:14.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:14.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:14.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:14.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:14.192+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:14.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:14.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:14.178+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:14.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:14.178+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:15.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:14.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:15.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:14.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:15.153 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:14.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:15.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:15.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:15.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:15.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:15.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:15.167+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:15.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:15.164+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:15.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:15.164+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:15.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:15.164+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:15.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:15.164+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:16.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:15.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:16.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:15.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:16.188 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:15.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:16.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:16.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:16.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:16.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:16.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:16.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:16.202+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:17.190 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:17.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:17.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:17.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:17.191 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:17.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:17.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:16.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:17.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:17.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:17.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:17.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:17.202+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:18.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:18.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:18.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:18.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:18.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:17.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:18.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:17.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:18.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:17.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:18.478 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:18.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:18.478 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:18.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:18.478 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:18.234+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:18.478 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:18.234+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:18.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:18.489+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27803 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:19.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:19.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:19.240 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:19.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:19.241 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:19.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:19.241 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:18.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:19.241 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:18.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:19.241 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:18.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:19.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:19.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:19.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:19.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:19.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:19.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:19.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:19.252+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:20.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:20.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:20.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:20.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:20.218 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:20.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:20.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:19.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:20.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:19.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:20.219 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:19.968+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:20.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:20.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:20.219 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:20.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:20.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:20.204+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:20.220 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:20.204+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:21.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:21.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:21.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:21.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:21.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:21.194+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:21.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:21.194+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:21.174+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:21.174+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:21.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:21.174+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:21.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:20.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:21.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:20.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:21.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:20.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:21.964+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:21.964+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:21.964+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:22.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:22.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:22.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:22.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:22.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:22.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:22.176+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:22.177+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:23.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:23.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:23.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:23.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:23.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:23.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:23.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:23.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:23.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:23.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:23.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:23.184+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:23.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:23.184+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:23.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:22.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:23.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:22.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:23.291 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:22.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:23.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:23.493+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27871 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:24.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:24.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:24.235 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:24.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:24.236 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:24.201+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:24.236 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:24.202+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:24.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:23.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:24.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:23.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:24.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:23.972+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:24.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:24.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:24.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:24.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:24.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:24.246+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:25.256 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:25.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:25.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:25.177+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:25.177+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:24.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:24.956+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:25.257 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:24.956+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:25.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:25.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:25.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:25.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:25.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:25.267+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:26.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:25.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:26.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:25.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:26.185 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:25.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:26.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:26.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:26.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:26.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:26.273+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:26.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:26.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:26.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:26.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:26.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:26.196+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:26.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:26.196+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:27.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:26.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:27.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:26.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:27.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:26.899+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:27.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:27.250+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:27.250+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:27.250+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:27.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:27.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:27.202+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:27.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:27.203+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:28.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:27.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:28.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:27.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:28.190 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:27.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:28.486 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:28.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:28.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:28.201+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:28.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:28.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:28.238+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:28.487 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:28.238+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:28.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:28.497+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 27939 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:29.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:28.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:29.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:28.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:29.186 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:28.849+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:29.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:29.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:29.194+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:29.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:29.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:29.235+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:29.235+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:30.168 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:29.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:30.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:29.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:30.169 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:29.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:30.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:30.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:30.180+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:30.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:30.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:30.263+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:30.263+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:31.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:31.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:31.178 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:31.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:31.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:31.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:31.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:31.219+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:31.219+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:31.220+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:31.220+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:31.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:31.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:31.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:32.165+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:32.165+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:32.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:32.165+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:32.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:32.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:32.171+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:32.171+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:33.132 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:32.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:33.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:32.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:33.133 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:32.836+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:33.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:33.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:33.491 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:33.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:33.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:33.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:33.491 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:33.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:33.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:33.221+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:33.492 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:33.221+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:33.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:33.502+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28008 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:34.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:33.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:34.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:33.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:34.167 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:33.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:34.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:34.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:34.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:34.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:34.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:34.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:34.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:34.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:34.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:34.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:34.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:34.188+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:34.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:34.188+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:35.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:34.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:35.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:34.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:35.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:34.786+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:35.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:35.203+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:35.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:35.203+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:35.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:35.203+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:35.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:35.203+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:35.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:35.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:35.139+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:35.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:35.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:36.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:35.798+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:36.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:36.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:36.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:36.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:36.192+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:36.193+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:36.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:36.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:36.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:36.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:36.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:36.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:36.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:37.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:37.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:37.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:37.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:37.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:37.122+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:37.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:37.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:37.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:37.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:37.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:37.160+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:37.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:37.160+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:37.748+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:37.748+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:37.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:37.748+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:38.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:38.208+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:38.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:38.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:38.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:38.209+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:38.495 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:38.209+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:38.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:38.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:38.496 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:38.123+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:38.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:38.506+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28082 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:38.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:38.746+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:38.746+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:38.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:38.746+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:39.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:39.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:39.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:39.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:39.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:39.076+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:39.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:39.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:39.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:39.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:39.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:39.252+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:39.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:39.252+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:40.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:39.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:40.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:39.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:40.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:39.778+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:40.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:40.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:40.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:40.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:40.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:40.127+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:40.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:40.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:40.301+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:40.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:40.301+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:41.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:40.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:41.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:40.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:41.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:40.782+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:41.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:41.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:41.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:41.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:41.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:41.121+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:41.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:41.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:41.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:41.321+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:41.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:41.321+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:41.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:41.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:41.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:42.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:42.154+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:42.347 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:42.154+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:42.348 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:42.154+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:42.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:42.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:42.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:42.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:42.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:42.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:42.739 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:42.359+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:42.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:42.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:43.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:42.750+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:43.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:43.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:43.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:43.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:43.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:43.347+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:43.499 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:43.347+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:43.499 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:43.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:43.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:43.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:43.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:43.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:43.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:43.510+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28155 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:43.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:43.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:43.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:43.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:43.758+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:44.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:44.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:44.157+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:44.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:44.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:44.395+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:44.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:44.395+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:45.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:44.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:45.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:44.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:45.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:44.774+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:45.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:45.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:45.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:45.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:45.426 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:45.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:45.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:45.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:45.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:45.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:45.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:45.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:45.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:45.437+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:45.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:45.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:46.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:45.773+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:46.454 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:46.209+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:46.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:46.209+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:46.455 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:46.209+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:46.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:46.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:46.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:46.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:46.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:46.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:46.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:46.466+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:47.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:46.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:47.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:46.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:47.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:46.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:47.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:47.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:47.504 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:47.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:47.446+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:47.446+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:47.446+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:47.505 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:47.446+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:47.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:47.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:47.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:47.747+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:48.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:48.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:48.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:48.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:48.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:48.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:48.752 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:48.514+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28232 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:48.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:48.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:48.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:48.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:48.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:48.493+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:48.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:48.493+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:49.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:48.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:48.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:48.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:49.111+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:49.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:49.111+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:49.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:49.111+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:49.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:49.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:49.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:49.741+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:49.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:49.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:49.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:49.494+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:50.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:50.071+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:50.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:50.071+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:50.466 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:50.071+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:50.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:50.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:50.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:50.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:50.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:50.706+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:50.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:50.477+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:50.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:50.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:50.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:50.478+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:50.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:50.478+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:51.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:51.035+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:51.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:51.035+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:51.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:51.035+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:51.733 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:51.449+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:51.733 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:51.449+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:51.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:51.449+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:51.734 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:51.449+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:51.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:51.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:51.745+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:52.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:52.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:52.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:52.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:52.396 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:52.060+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:52.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:52.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:52.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:52.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:52.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:52.407+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:52.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:52.407+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:52.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:52.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:53.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:52.749+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:53.369 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:53.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:53.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:53.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:53.370 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:53.074+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:53.634 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:53.519+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28310 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:53.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:53.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:53.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:53.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:53.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:53.381+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:53.634 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:53.381+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:53.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:53.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:53.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:54.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:54.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:54.117+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:54.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:54.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:54.363+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:54.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:54.363+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:55.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:55.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:55.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:54.753+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:55.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:55.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:55.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:55.371+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:55.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:55.371+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:55.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:55.143+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:55.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:55.143+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:55.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:55.143+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:55.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:55.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:56.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:55.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:56.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:56.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:56.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:56.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:56.365 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:56.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:56.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:56.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:56.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:56.376+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:56.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:56.376+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:56.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:56.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:57.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:56.734+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:57.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:57.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:57.500 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:57.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:57.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:57.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:57.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:57.353+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:57.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:57.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:57.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:57.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:57.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:57.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:57.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:57.735+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:58.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:58.322+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:58.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:58.322+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:58.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:58.322+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:58.512 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:58.322+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:47:58.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:58.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:58.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:58.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:58.513 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:58.160+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:58.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:47:58.523+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28387 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:47:58.789 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:58.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:58.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:58.790 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:58.764+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:47:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:59.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:59.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:59.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:47:59.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:47:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:59.350+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:47:59.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:47:59.350+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:59.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:59.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:00.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:47:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:47:59.770+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:00.481 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:00.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:00.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:00.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:00.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:00.224+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:00.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:00.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:00.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:00.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:00.482 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:00.320+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:00.483 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:00.320+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:01.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:00.754+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:01.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:01.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:01.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:01.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:01.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:01.190+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:01.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:01.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:01.352+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:01.352+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:01.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:01.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:01.725+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:02.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:02.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:02.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:02.212+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:02.387+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:02.387+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:02.387+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:02.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:02.387+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:02.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:03.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:02.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:03.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:02.765+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:03.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:03.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:03.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:03.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:03.516 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:03.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:03.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:03.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:03.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:03.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:03.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:03.379+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:03.517 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:03.379+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:03.768 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:03.527+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28464 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:04.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:03.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:04.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:03.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:04.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:03.781+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:04.128+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:04.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:04.128+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:04.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:04.128+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:04.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:04.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:04.356+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:04.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:04.356+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:05.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:04.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:05.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:04.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:05.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:04.763+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:05.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:05.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:05.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:05.162+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:05.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:05.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:05.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:05.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:05.388+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:06.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:05.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:06.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:05.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:06.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:05.787+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:06.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:06.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:06.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:06.189+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:06.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:06.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:06.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:06.358+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:06.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:06.358+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:07.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:06.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:07.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:06.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:07.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:06.808+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:07.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:07.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:07.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:07.501 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:07.330+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:07.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:07.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:07.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:07.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:07.502 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:07.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:08.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:07.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:08.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:07.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:08.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:07.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:08.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:08.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:08.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:08.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:08.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:08.330+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:08.521 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:08.330+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:08.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:08.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:08.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:08.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:08.525 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:08.141+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:08.783 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:08.532+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28541 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:09.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:08.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:09.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:08.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:09.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:08.795+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:09.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:09.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:09.364 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:09.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:09.364 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:09.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:09.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:09.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:09.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:09.374+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:09.365 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:09.374+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:10.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:09.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:10.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:09.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:10.165 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:09.817+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:10.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:10.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:10.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:10.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:10.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:10.176+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:10.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:10.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:10.364+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:10.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:10.364+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:11.150 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:10.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:11.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:10.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:11.151 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:10.833+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:11.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:11.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:11.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:11.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:11.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:11.408+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:11.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:11.408+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:11.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:11.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:11.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:11.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:11.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:11.163+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:11.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:11.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:11.821+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:12.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:12.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:12.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:12.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:12.418 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:12.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:12.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:12.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:12.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:12.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:12.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:12.429+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:12.770 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:12.430+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:13.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:12.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:13.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:12.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:13.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:12.831+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:13.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:13.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:13.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:13.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:13.377 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:13.105+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:13.649 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:13.536+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28618 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:13.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:13.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:13.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:13.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:13.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:13.388+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:13.650 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:13.388+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:14.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:13.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:14.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:13.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:14.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:13.870+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:14.398 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:14.055+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:14.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:14.055+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:14.399 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:14.055+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:14.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:14.821 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:14.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:14.821 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:14.410+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:14.822 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:14.410+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:15.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:14.839+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:15.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:14.839+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:15.091 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:14.839+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:15.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:15.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:15.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:15.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:15.373 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:15.103+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:15.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:15.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:15.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:15.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:15.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:16.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:15.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:16.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:15.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:16.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:15.888+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:16.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:16.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:16.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:16.101+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:16.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:16.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:16.415+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:16.415+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:17.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:16.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:17.128 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:16.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:17.129 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:16.844+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:17.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:17.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:17.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:17.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:17.413 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:17.140+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:17.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:17.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:17.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:17.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:17.424+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:18.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:17.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:18.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:17.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:18.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:17.890+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:18.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:18.138+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:18.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:18.138+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:18.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:18.138+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:18.653 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:18.541+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28696 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:18.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:18.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:18.653 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:18.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:18.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:18.390+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:18.654 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:18.390+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:19.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:18.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:19.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:18.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:19.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:18.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:19.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:19.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:19.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:19.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:19.291 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:19.131+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:19.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:19.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:19.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:19.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:19.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:19.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:19.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:19.428+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:20.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:20.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:20.222 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:20.137+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:19.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:20.222 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:19.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:20.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:19.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:20.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:20.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:20.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:20.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:20.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:20.384+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:20.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:20.384+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:21.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:20.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:21.156 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:20.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:21.157 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:20.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:21.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:21.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:21.168+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:21.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:21.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:21.400+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:21.400+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:21.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:21.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:21.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:22.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:22.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:22.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:22.159+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:22.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:22.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:22.373+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:22.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:22.373+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:22.545 ERROR:paramiko.transport:Socket exception: No route to host (113) 2023-12-19T19:48:22.550 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-19T19:48:22.551 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2023-12-19T19:48:22.552 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:48:22.567 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-19T19:48:22.579 INFO:tasks.cephadm:Teardown begin 2023-12-19T19:48:22.579 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:48:22.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-19T19:48:22.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:48:22.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:48:23.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:23.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:23.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:23.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:23.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:23.145+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:23.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:22.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:23.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:22.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:23.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:22.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:23.661 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:23.545+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28773 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:23.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:23.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:23.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:23.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:23.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:23.331+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:23.661 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:23.331+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:23.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:23.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:24.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:23.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:24.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:24.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:24.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:24.144+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:24.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:24.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:24.315+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:24.315+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:24.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:25.173+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:25.174+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:25.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:25.174+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:25.681 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:48:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:25.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:25.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:25.343+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:25.343+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:26.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:25.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:26.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:25.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:26.191 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:25.902+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:26.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:26.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:26.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:26.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:26.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:26.334+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:26.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:26.334+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:26.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:26.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:26.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:26.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:27.160 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:26.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:27.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:26.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:27.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:26.855+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:27.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:27.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:27.501 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:27.172+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:27.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:27.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:27.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:27.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:27.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:27.382+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:27.502 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:27.382+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:28.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:27.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:28.175 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:27.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:28.176 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:27.892+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:28.538 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:28.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:28.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:28.346+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:28.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:28.346+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:28.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:28.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:28.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:28.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:28.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:28.187+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:28.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:28.550+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28850 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:29.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:28.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:29.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:28.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:29.195 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:28.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:29.207+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:29.207+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:29.207+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:29.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:29.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:29.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:29.335+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:29.335+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:30.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:29.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:30.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:29.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:30.155 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:29.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:30.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:30.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:30.166+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:30.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:30.360+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:30.360+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:30.360+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:30.360+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:31.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:30.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:31.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:30.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:31.149 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:30.852+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:31.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:31.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:31.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:31.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:31.161+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:31.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:31.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:31.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:31.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:31.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:31.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:31.864+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:31.865+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:32.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:32.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:32.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:32.184+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:32.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:32.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:32.378+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:32.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:32.378+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:33.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:32.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:33.215 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:32.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:33.216 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:32.858+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:33.227+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:33.227+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:33.542 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:33.227+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:33.542 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:33.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:33.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:33.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:33.543 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:33.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:33.822 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:33.554+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 28928 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:34.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:33.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:34.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:33.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:34.250 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:33.834+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:34.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:34.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:34.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:34.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:34.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:34.398+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:34.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:34.398+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:34.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:34.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:34.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:34.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:34.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:34.261+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:34.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:48:34.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:48:35.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:34.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:35.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:34.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:35.252 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:34.813+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:35.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:35.263+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:35.263+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:35.263+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:35.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:35.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:35.437+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:35.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:35.437+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:36.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:35.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:36.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:35.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:36.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:35.838+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:36.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:36.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:36.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:36.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:36.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:36.284+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:36.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:36.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:36.440+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:36.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:36.440+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:37.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:36.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:37.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:36.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:37.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:36.876+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:37.745 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:48:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:37.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:37.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:37.323+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:37.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:37.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:37.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:37.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:37.451+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:37.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:37.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:37.925+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:38.672 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:38.559+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29005 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:38.672 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:38.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:38.672 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:38.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:38.673 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:38.349+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:38.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:38.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:38.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:38.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:38.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:38.402+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:38.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:38.402+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:38.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:38.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:38.930+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:39.350+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:39.350+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:39.350+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:39.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:39.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:39.442+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:39.443+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:39.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:39.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:40.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:39.910+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:40.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:40.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:40.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:40.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:40.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:40.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:40.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:40.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:40.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:40.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:40.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:40.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:40.541 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:40.444+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:40.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:40.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:40.931+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:41.753 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:41.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:41.753 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:41.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:41.753 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:41.369+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:41.753 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:41.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:41.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:41.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:41.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:41.417+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:41.754 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:41.417+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:41.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:41.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:41.962+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:42.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:42.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:42.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:42.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:42.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:42.374+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:42.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:42.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:42.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:42.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:42.441+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:42.441+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:42.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:42.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:42.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:43.676 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:43.563+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29082 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:43.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:43.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:43.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:43.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:43.677 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:43.342+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:43.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:43.677 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:43.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:43.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:43.678 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:43.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:43.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:43.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:44.287 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:43.945+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:44.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:44.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:44.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:44.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:44.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:44.466+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:44.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:44.466+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:44.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:44.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:44.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:44.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:44.299+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:44.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:44.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:44.901+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:45.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:45.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:45.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:45.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:45.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:45.348+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:45.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:45.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:45.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:45.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:45.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:46.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:46.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:46.518+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:46.518+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:46.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:46.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:46.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:47.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:46.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:47.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:46.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:47.273 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:46.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:47.530 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:47.285+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:47.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:47.285+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:47.531 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:47.285+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:47.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:47.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:47.542+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:47.542+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:48.264 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:47.919+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:48.265 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:47.919+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:48.265 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:47.919+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:48.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:48.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:48.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:48.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:48.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:48.276+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:48.869 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:48.567+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29159 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:48.869 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:48.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:48.869 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:48.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:48.869 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:48.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:48.870 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:48.552+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:49.246 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:48.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:49.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:48.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:49.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:48.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:49.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:49.258+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:49.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:49.258+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:49.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:49.258+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:49.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:48:49.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:48:49.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:49.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:49.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:49.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:49.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:49.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:49.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:49.551+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:50.223 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:49.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:49.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:50.224 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:49.903+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:50.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:50.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:50.257+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:50.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:50.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:50.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:50.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:50.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:50.595+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:50.904 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:50.595+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:51.235 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:50.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:51.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:50.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:51.236 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:50.915+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:51.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:51.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:51.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:51.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:51.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:51.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:51.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:51.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:51.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:51.934 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:51.589+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:52.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:51.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:52.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:51.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:52.260 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:51.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:52.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:52.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:52.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:52.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:52.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:52.272+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:52.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:48:52.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:52.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:52.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:52.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:52.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:52.578+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:52.981 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:52.578+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:52.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:52.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:52.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:53.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:53.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:53.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:53.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:53.560 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:53.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:54.025 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:53.572+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29237 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:54.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:53.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:54.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:53.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:54.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:53.596+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:54.026 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:53.596+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:54.275 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:54.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:54.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:54.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:54.276 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:54.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:54.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:54.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:54.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:54.287+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:54.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:54.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:54.569+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:54.569+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:55.514 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:55.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:55.515 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:55.305+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:55.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:55.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:55.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:55.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:55.515 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:55.065+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:55.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:55.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:55.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:55.527+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:55.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:55.527+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:55.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:48:55.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:48:55.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:48:55.894 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:48:55.894 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-19T19:48:55.894 DEBUG:teuthology.orchestra.run.smithi055:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-19T19:48:55.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:48:55.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:48:56.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:56.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:56.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.033+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:56.314+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:56.314+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:56.314+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:56.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:56.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:56.489+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:56.489+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:57.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:57.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:57.263 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:56.999+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:57.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:57.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:57.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:57.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:57.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:57.274+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:57.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:57.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:57.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:57.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:57.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:57.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:57.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:57.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:58.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:58.231+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:58.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:58.231+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:58.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:58.231+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:58.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:57.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:58.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:57.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:58.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:57.990+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:58.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:48:58.576+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29314 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:48:58.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:58.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:58.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:58.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:58.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:58.532+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:58.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:58.532+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:48:58.997 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:48:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:59.271+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:59.271+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:59.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:48:59.271+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:48:59.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:59.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:59.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:59.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:59.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:48:59.030+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:48:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:59.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:59.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:59.491+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:48:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:48:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:48:59.491+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:00.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:00.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:00.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:00.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:00.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:00.242+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:00.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:00.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:00.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:00.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:00.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:00.041+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:00.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:00.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:00.454+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:00.454+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:01.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:01.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:01.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:01.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:01.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:01.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:01.058+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:01.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:01.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:01.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:01.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:01.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:01.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:01.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:01.499+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:02.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:02.216+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:02.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:02.216+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:02.458 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:02.216+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:02.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:02.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:02.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:02.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:02.459 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:02.069+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:02.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:02.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:02.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:02.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:02.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:02.469+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:02.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:02.469+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:03.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:03.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:03.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:03.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:03.436 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:03.105+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:03.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:03.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:03.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:03.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:03.437 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:03.169+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:03.694 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:03.580+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29392 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:03.694 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:03.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:03.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:03.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:03.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:03.447+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:03.695 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:03.448+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:04.482 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:04.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:04.483 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:04.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:04.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:04.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:04.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:04.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:04.483 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:04.144+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:04.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:04.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:04.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:04.494+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:05.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:05.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:05.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:05.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:05.457 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:05.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:05.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:05.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:05.457 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:05.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:05.458 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:05.131+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:05.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:05.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:05.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:05.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:05.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:05.468+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:05.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:05.469+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:06.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:06.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:06.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:06.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:06.433 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:06.182+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:06.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:06.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:06.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:06.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:06.434 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:06.114+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:06.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:06.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:06.444+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:06.445+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:07.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:07.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:07.473 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:07.199+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:07.071+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:07.071+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:07.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:07.071+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:07.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:07.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:07.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:07.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:07.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:07.484+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:07.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:07.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:07.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:07.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:08.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:08.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:08.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:08.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:08.500 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:08.200+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:08.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:08.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:08.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:08.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:08.501 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:08.053+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:08.789 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:08.585+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29470 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:08.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:08.511+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:08.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:08.511+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:08.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:08.511+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:08.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:08.512+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:09.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:09.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:09.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:09.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:09.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:09.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:09.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:09.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:09.290 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:09.202+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:09.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:09.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:09.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:09.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:09.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:09.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:09.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:09.523+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:10.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:10.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:10.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:10.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:10.509 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:10.052+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:10.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:10.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:10.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:10.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:10.510 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:10.178+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:10.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:10.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:10.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:10.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:10.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:10.521+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:10.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:10.521+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:11.057 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:11.473 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:11.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:11.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:11.474 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:11.100+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:11.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:11.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:11.474 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:11.210+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:11.755 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:11.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:11.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:11.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:11.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:11.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:11.756 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:11.485+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:12.459 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:12.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:12.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:12.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:12.460 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:12.248+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:12.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:12.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:12.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:12.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:12.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:12.080+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:12.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:12.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:12.471+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:12.471+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:13.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:13.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:13.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:13.234+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:13.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:13.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:13.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:13.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:13.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:13.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:13.703 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:13.589+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29549 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:13.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:13.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:13.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:13.421+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:13.704 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:13.421+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:14.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:14.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:14.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:14.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:14.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:14.222+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:14.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:14.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:14.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:14.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:14.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:14.414+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:14.414+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:15.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:15.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:15.161 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:14.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:15.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:15.255+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:15.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:15.255+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:15.416 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:15.255+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:15.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:15.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:15.428+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:15.428+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:16.242 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:15.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:16.243 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:15.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:16.243 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:15.932+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:16.254+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:16.254+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:16.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:16.254+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:16.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:16.438+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:16.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:16.438+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:17.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:17.230+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:17.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:17.230+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:17.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:17.230+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:17.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:16.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:17.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:16.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:17.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:16.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:17.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:17.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:17.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:17.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:18.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:17.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:18.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:17.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:18.200 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:17.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:18.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:18.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:18.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:18.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:18.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:18.211+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:18.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:18.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:18.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:18.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:18.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:18.412+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:18.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:18.412+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:18.979 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:18.594+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29627 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:19.244 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:18.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:19.244 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:18.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:19.244 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:18.991+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:19.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:19.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:19.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:19.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:19.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:19.256+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:19.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:19.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:19.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:19.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:19.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:19.394+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:19.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:19.394+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:20.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:19.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:20.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:19.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:20.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:19.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:20.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:20.269+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:20.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:20.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:20.436+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:20.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:20.436+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:21.247 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:20.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:21.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:20.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:21.248 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:20.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:21.259+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:21.259+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:21.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:21.259+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:21.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:21.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:21.430+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:21.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:21.430+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:21.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:21.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:21.955+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:22.303+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:22.303+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:22.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:22.303+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:22.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:22.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:22.472+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:22.472+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:23.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:23.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:23.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:22.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:23.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:22.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:23.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:22.921+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:23.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:23.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:23.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:23.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:23.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:23.424+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:23.587 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:23.424+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:23.587 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:23.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:23.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:23.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:23.588 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:23.318+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:23.871 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:23.598+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29705 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:23.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:23.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:23.883+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:24.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:24.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:24.334+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:24.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:24.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:24.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:24.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:24.928+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:24.928+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:24.928+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:25.768 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:25.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:25.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:25.360+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:25.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:25.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:25.406+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:25.769 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:25.406+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:26.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:25.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:26.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:25.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:26.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:25.942+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:26.129 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:26.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:26.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:26.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:26.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:26.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:26.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:26.423+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:26.423+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:27.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:27.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:27.251 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:26.948+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:27.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:27.282+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:27.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:27.282+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:27.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:27.282+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:27.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:27.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:27.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:27.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:27.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:27.464+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:27.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:27.464+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:28.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:27.907+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:28.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:27.907+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:28.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:27.907+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:28.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:28.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:28.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:28.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:28.591 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:28.302+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:28.591 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:28.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:28.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:28.450+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:28.592 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:28.450+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:28.847 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:28.602+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29783 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:29.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:29.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:29.201 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:29.201 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:49:29.202 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-19T19:49:29.204 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329/remote/smithi055/crash 2023-12-19T19:49:29.205 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/crash -- . 2023-12-19T19:49:29.239 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:28.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:29.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:28.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:29.240 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:28.859+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:29.259 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329/remote/smithi187/crash 2023-12-19T19:49:29.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:29.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:29.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:29.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:29.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:29.330+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:29.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:29.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:29.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:29.433+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:29.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:29.433+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:30.226 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:29.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:30.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:29.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:30.227 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:29.825+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:30.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:30.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:30.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:30.358+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:30.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:30.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:30.476+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:30.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:30.476+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:31.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:31.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:31.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:30.823+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:31.757 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:31.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:31.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:31.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:31.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:31.396+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:31.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:31.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:31.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:31.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:31.758 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:31.498+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:31.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:31.498+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:31.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:31.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:32.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:31.867+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:32.341 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:32.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:32.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:32.442+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:32.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:32.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:32.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:32.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:32.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:33.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:32.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:33.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:32.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:33.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:32.881+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:33.723 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:33.607+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29862 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:33.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:33.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:33.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:33.449+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:33.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:33.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:33.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:33.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:33.724 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:33.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:34.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:33.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:34.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:33.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:34.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:33.913+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:34.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:34.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:34.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:34.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:34.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:34.475+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:34.475+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:35.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:34.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:35.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:34.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:35.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:34.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:35.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:35.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:35.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:35.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:35.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:35.494+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:35.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:35.494+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:35.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:35.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:35.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:35.382+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:36.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:35.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:36.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:35.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:36.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:35.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:36.737 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:36.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:36.737 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:36.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:36.737 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:36.391+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:36.737 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:36.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:36.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:36.460+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:36.738 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:36.460+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:37.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:36.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:37.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:36.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:37.040 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:36.986+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:37.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:37.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:37.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:37.394+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:37.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:37.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:37.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:37.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:37.435+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:37.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:37.435+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:37.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:37.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:38.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:37.973+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:38.722 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:38.611+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 29940 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:38.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:38.722 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:38.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:38.723 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:38.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:38.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:38.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:38.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:38.723 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:38.483+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:38.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:38.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:39.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:38.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:39.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:39.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:39.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:39.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:39.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:39.505+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:39.506+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:40.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:40.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:40.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.005+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:40.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:40.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:40.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:40.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:40.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:40.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:40.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:40.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:40.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:40.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:40.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:40.529+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:40.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:40.529+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:40.982+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:41.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:41.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:41.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:41.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:41.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:41.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:41.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:41.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:41.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:41.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:41.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:41.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:41.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:41.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:41.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:41.503+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:41.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:41.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:42.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:41.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:42.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:42.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:42.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:42.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:42.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:42.419+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:42.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:42.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:42.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:42.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:42.492+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:42.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:42.492+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:42.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:42.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:43.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:42.926+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:43.731 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:43.615+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30018 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:43.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:43.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:43.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:43.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:43.731 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:43.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:43.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:43.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:43.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:43.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:43.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:43.522+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:43.732 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:43.522+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:44.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:43.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:44.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:43.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:44.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:43.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:44.401 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:44.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:44.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:44.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:44.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:44.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:44.393+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:44.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:44.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:44.501+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:44.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:44.501+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:45.008+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:45.008+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:45.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:45.008+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:45.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:45.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:45.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:45.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:45.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:45.418+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:45.513+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:45.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:45.513+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:46.018+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:46.018+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:46.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:46.018+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:46.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:46.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:46.503+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:46.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:46.503+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:46.372+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:46.372+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:46.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:46.372+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:47.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:47.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:47.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:47.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:47.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:47.037+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:47.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:47.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:47.483+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:47.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:47.483+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:47.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:47.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:47.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:47.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:47.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:47.398+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:48.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:48.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:48.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.002+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:48.621 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:48.620+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30096 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:48.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:48.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:48.375+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:48.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:48.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:48.434+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:48.622 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:48.436+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:49.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:49.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:49.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:48.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:49.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:49.400+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:49.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:49.400+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:49.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:49.400+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:49.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:49.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:49.451+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:49.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:49.451+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:50.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:49.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:50.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:49.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:50.228 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:49.946+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:50.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:50.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:50.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:50.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:50.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:50.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:50.485+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:50.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:50.486+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:51.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:50.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:51.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:50.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:51.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:50.981+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:51.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:51.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:51.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:51.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:51.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:51.475+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:51.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:51.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:51.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:51.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:51.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:51.499+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:51.760 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:51.499+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:51.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:51.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:52.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:51.935+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:52.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:52.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:52.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:52.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:52.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:52.471+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:52.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:52.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:52.510+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:52.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:52.510+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:52.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:52.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:52.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:53.740 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:53.624+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30175 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:53.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:53.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:53.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:53.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:53.740 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:53.428+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:53.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:53.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:53.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:53.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:53.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:53.507+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:53.741 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:53.507+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:53.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:53.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:54.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:53.947+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:54.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:54.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:54.536 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:54.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:54.537 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:54.401+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:54.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:54.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:54.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:54.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:54.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:54.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:54.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:54.548+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:55.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:54.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:55.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:54.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:55.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:54.992+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:55.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:55.402+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:55.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:55.402+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:55.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:55.402+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:55.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:55.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:55.580+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:55.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:55.580+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:56.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:56.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:56.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:56.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:56.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:56.032+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:56.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:49:56.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:49:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:56.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:56.422+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:56.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:56.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:56.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:56.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:56.587+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:57.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:57.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:57.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.014+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:57.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:57.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:57.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:57.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:57.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:57.427+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:57.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:57.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:57.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:57.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:57.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:57.599+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:57.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:57.600+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:58.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:58.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:58.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:57.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:58.638 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:49:58.629+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30253 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:49:58.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:58.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:58.638 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:58.388+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:58.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:58.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:58.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:58.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:58.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:58.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:58.639 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:58.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:59.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:58.976+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:59.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:58.976+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:59.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:58.976+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:49:59.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:49:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:59.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:59.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:59.528+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:49:59.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:49:59.528+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:49:59.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:59.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:59.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:59.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:49:59.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:49:59.415+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:00.229 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:59.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:00.229 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:59.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:00.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:49:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:49:59.940+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:00.436+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:00.436+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:00.436+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:00.932 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:00.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:00.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:00.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:00.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:00.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:00.933 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:00.563+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:01.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:00.944+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:01.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:00.944+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:01.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:00.944+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:01.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:01.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:01.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:01.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:01.761 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:01.420+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:01.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:01.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:01.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:01.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:01.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:01.547+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:01.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:01.547+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:01.937+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:01.937+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:01.937+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:02.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:50:02.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:50:02.549 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:50:02.550 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:50:02.550 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-19T19:50:02.550 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-19T19:50:02.591 INFO:teuthology.orchestra.run.smithi055.stderr:grep: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-19T19:50:02.594 INFO:tasks.cephadm:Compressing logs... 2023-12-19T19:50:02.594 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T19:50:02.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:50:02.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:50:02.668 INFO:teuthology.orchestra.run.smithi055.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2023-12-19T19:50:02.669 INFO:teuthology.orchestra.run.smithi055.stderr: -- /var/log/ceph/cephadm.log 2023-12-19T19:50:02.669 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.wupdfj.log 2023-12-19T19:50:02.669 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-19T19:50:02.670 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.qvunsw.log 2023-12-19T19:50:02.670 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.wupdfj.log: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-19T19:50:02.671 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.qvunsw.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mon.smithi055.log 2023-12-19T19:50:02.671 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mgr.smithi055.dwclvp.log 2023-12-19T19:50:02.672 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mon.smithi055.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-19T19:50:02.672 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mgr.smithi055.dwclvp.log: gzip -5 --verbose -- /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-19T19:50:02.678 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-19T19:50:02.686 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-19T19:50:02.704 INFO:teuthology.orchestra.run.smithi055.stderr: 95.7% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.qvunsw.log.gz 2023-12-19T19:50:02.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:02.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:02.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:02.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:02.791 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:02.446+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:02.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:02.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:02.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:02.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:02.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:02.546+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:02.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:02.546+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:03.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:02.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:03.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:02.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:03.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:02.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:03.448 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.0.log: file size changed while zipping 2023-12-19T19:50:03.449 INFO:teuthology.orchestra.run.smithi055.stderr: 92.7% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-19T19:50:03.560 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mgr.smithi055.dwclvp.log: file size changed while zipping 2023-12-19T19:50:03.560 INFO:teuthology.orchestra.run.smithi055.stderr: 89.9% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mgr.smithi055.dwclvp.log.gz 2023-12-19T19:50:03.625 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:03.632+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30330 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:03.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:03.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:03.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:03.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:03.625 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:03.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:03.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:03.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:03.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:03.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:03.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:03.560+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:03.626 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:03.560+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:03.629 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mon.smithi055.log: file size changed while zipping 2023-12-19T19:50:03.629 INFO:teuthology.orchestra.run.smithi055.stderr: 92.5% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mon.smithi055.log.gz 2023-12-19T19:50:03.671 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.2.log: file size changed while zipping 2023-12-19T19:50:03.672 INFO:teuthology.orchestra.run.smithi055.stderr: 92.8% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-19T19:50:03.814 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.1.log: file size changed while zipping 2023-12-19T19:50:03.814 INFO:teuthology.orchestra.run.smithi055.stderr: 92.8% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-19T19:50:04.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:03.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:04.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:03.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:04.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:03.923+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:04.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:04.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:04.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:04.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:04.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:04.386+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:04.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:04.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:04.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:04.568+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:04.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:04.568+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:05.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:04.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:05.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:04.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:05.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:04.898+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:05.717 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:50:05.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:05.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:05.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:05.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:05.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:05.347+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:05.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:05.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:05.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:05.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:05.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:05.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:06.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:05.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:06.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:05.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:06.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:05.856+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:06.788 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:06.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:06.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:06.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:06.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:06.355+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:06.591+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:06.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:06.591+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:06.591+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:06.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:06.591+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:07.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:07.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:07.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:06.894+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:07.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:07.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:07.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:07.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:07.629 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:07.384+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:07.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:07.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:07.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:07.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:07.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:07.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:07.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:07.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:07.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:07.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:07 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:07.920+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:08.746 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:08.635+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30401 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:08.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:08.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:08.746 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:08.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:08.747 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:08.433+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:08.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:08.622+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:08.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:08.622+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:08.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:08.622+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:08.747 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:08.622+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:09.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:08.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:09.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:08.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:09.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:08 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:08.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:09.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:09.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:09.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:09.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:09.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:09.395+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:09.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:09.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:09.640+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:09.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:09.640+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:10.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:09.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:10.230 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:09.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:10.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:09 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:09.917+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:10.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:10.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:10.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:10.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:10.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:10.445+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:10.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:10.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:10.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:10.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:10.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:10.654+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:10.947 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:10.654+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:11.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:10.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:11.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:10.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:11.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:10 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:10.959+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:11.760 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:11.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:11.761 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:11.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:11.761 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:11.421+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:11.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:11.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:11.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:11.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:11.761 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:11.670+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:11.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:11.670+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:11.923 INFO:teuthology.orchestra.run.smithi055.stderr: 92.6% -- replaced with /var/log/ceph/63de9ba6-9ea1-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi055.wupdfj.log.gz 2023-12-19T19:50:11.924 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-12-19T19:50:11.925 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m9.274s 2023-12-19T19:50:11.925 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m13.735s 2023-12-19T19:50:11.925 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.347s 2023-12-19T19:50:12.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:11.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:11.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:12.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:11 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:11.957+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:12.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:12.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:12.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:12.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:12.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:12.621+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:12.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:12.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:12.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:12.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:12.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:12.432+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:13.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:12.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:13.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:12.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:13.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:12 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:12.912+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:13.751 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:13.640+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30479 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:13.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:13.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:13.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:13.423+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:13.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:13.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:13.627+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:13.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:13.627+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:14.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:13.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:14.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:13.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:14.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:13 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:13.893+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:14.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:50:14.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:50:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:14.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:14.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:14.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:14.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:14.642+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:14.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:14.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:14.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:14.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:14.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:14.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:15.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:14.927+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:15.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:14.927+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:15.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:14 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:14.927+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:15.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:15.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:15.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:15.636+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:15.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:15.636+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:15.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:15.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:15.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:15.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:15.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:15.411+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:16.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:15.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:16.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:15.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:16.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:15 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:15.953+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:16.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:16.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:16.645+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:16.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:16.645+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:16.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:16.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:16.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:16.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:16.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:16.441+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:17.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:16.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:17.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:16.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:17.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:16 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:16.966+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:17.781 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:50:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:17.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:17.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:17.648+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:17.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:17.648+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:17.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:17.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:17.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:17.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:17.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:17.406+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:18.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:17.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:18.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:17.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:18.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:17 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:17.943+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:18.758 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:18.645+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30557 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:18.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:18.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:18.758 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:18.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:18.759 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:18.425+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:18.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:18.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:18.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:18.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:18.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:18.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:18.759 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:18.602+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:19.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:18.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:19.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:18.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:19.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:18 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:18.941+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:19.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:19.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:19.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:19.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:19.790 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:19.438+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:19.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:19.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:19.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:19.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:19.791 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:19.625+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:19.792 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:19.625+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:20.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:19.979+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:20.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:19.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:20.231 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:19 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:19.980+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:20.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:20.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:20.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:20.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:20.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:20.412+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:21.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:20.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:20.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.001 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:20.607+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.002 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:20 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:20.607+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:21.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.013+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:21.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.013+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:21.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.013+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:21.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:21.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:21.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:21.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:21.762 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:21.443+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:21.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:21.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.762 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:21.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.763 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:21.602+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:21.763 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:21.602+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:22.038 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.996+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.996+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:22.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:21 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:21.996+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:22.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:22.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:22.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:22.481+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:22.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:22.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:22.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:22.589+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:22.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:22 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:22.589+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:23.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:23.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:23.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:23.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:23.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:23.040+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:23.762 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:23.649+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30635 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:23.763 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:23.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:23.763 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:23.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:23.763 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:23.510+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:23.763 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:23.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:23.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:23.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:23.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:23.620+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:23.764 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:23 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:23.620+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:24.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:24.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:24.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:24.038+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:24.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:24.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:24.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:24.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:24.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:24.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:24.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:24.575+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:24.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:24 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:24.575+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:25.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:25.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:25.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:25.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:25.043+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:25.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:25.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:25.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:25.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:25.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:25.539+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:25.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:25.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:25.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:25.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:25.558+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:25.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:25 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:25.558+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:26.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:26.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:26.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:26.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:26.530 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:26.077+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:26.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:26.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:26.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:26.542+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:26.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:26.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:26.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:26.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:26.551+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:26.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:26 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:26.551+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:27.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:27.126+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:27.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:27.126+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:27.482 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:27.126+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:27.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:27.494+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:27.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:27.494+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:27.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:27.494+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:27.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:27.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:27.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:27.552+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:27.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:27 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:27.552+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:28.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:28.159+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:28.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:28.159+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:28.502 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:28.159+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:28.766 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:28.653+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30713 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:28.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:28.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:28.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:28.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:28.767 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:28.514+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:28.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:28.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:28.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:28.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:28.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:28.550+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:28.768 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:28 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:28.550+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:29.503 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:29.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:29.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:29.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:29.504 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:29.192+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:29.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:50:29.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:50:29.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:29.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:29.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:29.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:29.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:29.515+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:29.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:29.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:29.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:29.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:29.531+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:29.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:29 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:29.531+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:30.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:30.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:30.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:30.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:30.508 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:30.157+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:30.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:30.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:30.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:30.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:30.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:30.521+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:30.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:30.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:30.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:30.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:30.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:30.563+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:30.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:30 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:30.563+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:31.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:31.137+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:31.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:31.137+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:31.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:31.137+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:31.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:31.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:31.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:31.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:31.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:31.565+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:31.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:31.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:31.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:31.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:31.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:31.537+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:31.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:31 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:31.537+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:32.485 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:32.179+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:32.485 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:32.179+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:32.486 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:32.179+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:32.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:32.577+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:32.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:32.497+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:32.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:32.497+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:32.497+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:32.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:32 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:32.497+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:32.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:50:33.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:33.206+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:33.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:33.206+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:33.511 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:33.206+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:33.771 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:33.657+0000 7f7e92e41700 -1 mon.smithi055@0(probing) e2 get_health_metrics reporting 30792 slow ops, oldest is monmgrreport(gid 14208, 0 checks, 1 progress events) 2023-12-19T19:50:33.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:33.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:33.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:33.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:33.772 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:33.557+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:33.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:33.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:33.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:33.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:33.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:33.523+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:33.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:33 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:33.523+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:34.525 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:34.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:34.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:34.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:34.526 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:34.233+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:34.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:34.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:34.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:34.598+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:34.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:34.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:34.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:34.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:34.538+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:34.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:34 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:34.538+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:35.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:35.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:35.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:35.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:35.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:35.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:35.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:50:35.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:50:35.921 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:50:35.922 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:50:35.922 INFO:tasks.cephadm:Removing cluster... 2023-12-19T19:50:35.922 DEBUG:teuthology.orchestra.run.smithi055:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 --force 2023-12-19T19:50:35.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:35.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:35.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:35.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:35.956 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:35.626+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:35.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:35.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:35.956 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:35.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:35.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:35.586+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:35.957 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:35 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:35.586+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:36.467 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:36 smithi055 systemd[1]: Stopping Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:50:36.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:36.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:36.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:36.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:36.468 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:36.218+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:36.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:36.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:36.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:36.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:36.773 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:36.618+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:36.774 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:36.618+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:36.774 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:36.479+0000 7f7e95e47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi055 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:50:36.774 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055[80974]: 2023-12-19T19:50:36.479+0000 7f7e95e47700 -1 mon.smithi055@0(probing) e2 *** Got Signal Terminated *** 2023-12-19T19:50:36.774 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:36.628+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:36.774 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:36.628+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:36.775 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:36 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:36.628+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:37.038 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:36 smithi055 podman[136858]: 2023-12-19 19:50:36.785825501 +0000 UTC m=+0.414856493 container died 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3) 2023-12-19T19:50:37.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 podman[136858]: 2023-12-19 19:50:37.180278328 +0000 UTC m=+0.809309304 container cleanup 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2023-12-19T19:50:37.335 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 bash[136858]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055 2023-12-19T19:50:37.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:37.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:37.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:37.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:37.336 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:37.241+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:37.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 podman[136872]: 2023-12-19 19:50:37.3485949 +0000 UTC m=+0.562680827 container remove 22d858bf8f4418a2b8e1a9cb555857f22a0c55c4a00b865c1356e12bcd797d7f (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-mon-smithi055, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2) 2023-12-19T19:50:37.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service: Deactivated successfully. 2023-12-19T19:50:37.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 systemd[1]: Stopped Ceph mon.smithi055 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:50:37.586 INFO:journalctl@ceph.mon.smithi055.smithi055.stdout:Dec 19 19:50:37 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@mon.smithi055.service: Consumed 27.216s CPU time. 2023-12-19T19:50:38.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:37.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.014 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:37.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:37.658+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.015 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:37.658+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:38.015 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:37.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:38.015 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:37.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:38.015 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:37 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:37.601+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:38.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:38.281+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:38.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:38.281+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:38.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:38.281+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:38.816 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:38.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.816 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:38.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.816 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:38.628+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:38.816 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:38.628+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:38.817 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:38.632+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:38.817 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:38.632+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:38.817 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:38 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:38.632+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:39.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:39.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:39.445 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:39.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:39.446 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:39.279+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:39.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:39.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:39.789 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:39.584+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:39.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:39.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:39.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:39.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:39.631+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:39.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:39 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:39.631+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:40.488 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:40.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:40.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:40.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:40.489 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:40.231+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:40.751 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:40.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:40.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:40.603+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:40.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:40.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:40.606+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:40.752 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:40 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:40.606+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:41.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:41.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:41.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:41.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:41.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:41.609+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:41.609+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:41.724 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:41.609+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:41.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:41.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:41.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:41.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:41.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:41.619+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:41.725 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:41 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:41.619+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:42.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:42.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:42.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:42.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:42.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:42.225+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:42.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:42.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:42.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:42.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:42.909 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:42.655+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:42.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:42.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:42.909 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:42.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:42.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:42.582+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:42.910 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:42 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:42.582+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:43.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:43.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:43.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:43.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:43.460 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:43.177+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:43.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:43.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:43.777 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:43.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:43.778 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:43.668+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:43.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:43.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:43.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:43.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:43.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:43.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:43.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:43 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:43.587+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:44.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:44.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:44.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:44.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:44.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:44.191+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:44.906 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:44.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:44.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:44.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:44.907 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:44.667+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:44.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:44.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:44.907 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:44.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:44.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:44.587+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:44.908 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:44 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:44.587+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:45.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:45.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:45.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:45.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:45.203 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:45.152+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:45.538 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 systemd[1]: Stopping Ceph osd.0 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:50:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.464+0000 7f8860a7d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:50:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.464+0000 7f8860a7d700 -1 osd.0 46 *** Got signal Terminated *** 2023-12-19T19:50:45.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.464+0000 7f8860a7d700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:50:46.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:46.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:46.040 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:45.715+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:45.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:45.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:45.603+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:46.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:45 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:45.603+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:46.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:46.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:46.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:46.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:46.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:46.187+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:47.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:46.726+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:47.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:46.726+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:47.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:46.726+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:47.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:46.646+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:47.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:46 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:46.646+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:47.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:47.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:47.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:47.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:47.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:47.227+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:48.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:47.721+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:48.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:47.721+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:48.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:47.721+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:48.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:47.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:48.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:47.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:48.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:47.638+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:48.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:47 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:47.638+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:48.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:48.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:48.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:48.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:48.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:48.274+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:49.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:48.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:49.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:48.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:49.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:48.756+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:49.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:48.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:49.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:48.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:49.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:48.664+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:49.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:48 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:48.664+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:49.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:49.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:49.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:49.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:49.609 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:49.323+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:50.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:49.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:50.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:49.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:50.039 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0[99783]: 2023-12-19T19:50:49.746+0000 7f885b077700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:16.473239+0000 (oldest deadline 2023-12-19T19:45:36.473239+0000) 2023-12-19T19:50:50.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:49.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:50.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:49.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:50.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:49.621+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:50.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:49 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:49.621+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:50.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:50.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:50.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:50.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:50.651 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:50.330+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:51.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:50.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:50.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:50.663+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.007 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:50 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:50.663+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:51.272 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 podman[137714]: 2023-12-19 19:50:51.019418537 +0000 UTC m=+5.663417538 container died b82fbcd42bf04dcd47213a632e1c2bfe4e3b57da7dd799e0041fe224802dff9f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:50:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 podman[137714]: 2023-12-19 19:50:51.381553692 +0000 UTC m=+6.025552659 container cleanup b82fbcd42bf04dcd47213a632e1c2bfe4e3b57da7dd799e0041fe224802dff9f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2023-12-19T19:50:51.539 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 bash[137714]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0 2023-12-19T19:50:51.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:51.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:51.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:51.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:51.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:51.284+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:51.979 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 podman[137733]: 2023-12-19 19:50:51.657709691 +0000 UTC m=+0.628016293 container remove b82fbcd42bf04dcd47213a632e1c2bfe4e3b57da7dd799e0041fe224802dff9f (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0, org.label-schema.vendor=CentOS, RELEASE=quincy-a88f3ed, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2) 2023-12-19T19:50:51.979 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:51.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.979 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:51.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:51.675+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:51.980 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:51 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:51.675+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:52.255 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 podman[137828]: 2023-12-19 19:50:51.992006288 +0000 UTC m=+0.114817112 container create c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, io.buildah.version=1.31.3, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, ceph=True) 2023-12-19T19:50:52.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:51 smithi055 podman[137828]: 2023-12-19 19:50:51.895027946 +0000 UTC m=+0.017838789 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:50:52.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:52 smithi055 podman[137828]: 2023-12-19 19:50:52.211238656 +0000 UTC m=+0.334049499 container init c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2023-12-19T19:50:52.256 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:52 smithi055 podman[137828]: 2023-12-19 19:50:52.214571635 +0000 UTC m=+0.337382469 container start c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, io.buildah.version=1.31.3, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2023-12-19T19:50:52.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:52 smithi055 podman[137828]: 2023-12-19 19:50:52.294521105 +0000 UTC m=+0.417331943 container attach c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-a88f3ed, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2023-12-19T19:50:52.522 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:52 smithi055 podman[137828]: 2023-12-19 19:50:52.507862476 +0000 UTC m=+0.630673307 container died c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-a88f3ed) 2023-12-19T19:50:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:52.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:52.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:52.523 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:52.267+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:52.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:52.678+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:52.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:52.678+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:52.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:52.678+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:52.778 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:52 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:52.678+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:53.029 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:53 smithi055 podman[137854]: 2023-12-19 19:50:53.039707966 +0000 UTC m=+0.526626383 container remove c332eb383d4a9ecf7bcd520f9c6b5f9648ea51424264a2d6ef1763a656bf85c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-a88f3ed, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2) 2023-12-19T19:50:53.288 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 systemd[1]: Stopping Ceph osd.1 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:50:53.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:53 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.0.service: Deactivated successfully. 2023-12-19T19:50:53.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:53 smithi055 systemd[1]: Stopped Ceph osd.0 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:50:53.289 INFO:journalctl@ceph.osd.0.smithi055.stdout:Dec 19 19:50:53 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.0.service: Consumed 16.922s CPU time. 2023-12-19T19:50:53.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:53.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:53.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:53.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:53.290 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:53.273+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:53.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.520+0000 7fa133002700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:50:53.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.520+0000 7fa133002700 -1 osd.1 46 *** Got signal Terminated *** 2023-12-19T19:50:53.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.520+0000 7fa133002700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:50:53.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:53.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.642+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:53.790 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:53 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:53.642+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:54.538 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:54.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:54.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:54.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:54.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:54.236+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:54.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:54.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:54.649+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:55.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:54 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:54.649+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:55.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:55.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:55.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:55.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:55.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:55.194+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:56.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:55.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:56.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:55.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:56.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:55.605+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:56.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:55 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:55.605+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:56.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:56.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:56.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:56.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:56.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:56.234+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:57.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:56.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:57.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:56.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:56.567+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:57.040 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:56 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:56.567+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:57.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:57.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:57.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:57.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:57.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:57.224+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:58.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:57.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:58.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:57.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:58.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:57.554+0000 7fa130e03700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:19.384562+0000 (oldest deadline 2023-12-19T19:45:39.384562+0000) 2023-12-19T19:50:58.039 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:57 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1[106934]: 2023-12-19T19:50:57.554+0000 7fa130e03700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.4:287 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=85286b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2023-12-19T19:50:58.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:58.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:58.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:58.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:58.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:58 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:58.253+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:59.315 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:59 smithi055 podman[137944]: 2023-12-19 19:50:59.0679857 +0000 UTC m=+5.668862560 container died 0e674f452739ed5978628025906cf37a7633924fd8a1439b37abd1f9f6513b9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS) 2023-12-19T19:50:59.315 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:59.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:59.315 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:59.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:59.316 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:50:59 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:50:59.275+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:50:59.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:59 smithi055 podman[137944]: 2023-12-19 19:50:59.426606819 +0000 UTC m=+6.027483671 container cleanup 0e674f452739ed5978628025906cf37a7633924fd8a1439b37abd1f9f6513b9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed) 2023-12-19T19:50:59.673 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:59 smithi055 bash[137944]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1 2023-12-19T19:51:00.030 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:50:59 smithi055 podman[137961]: 2023-12-19 19:50:59.686109471 +0000 UTC m=+0.611476548 container remove 0e674f452739ed5978628025906cf37a7633924fd8a1439b37abd1f9f6513b9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-a88f3ed, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:51:00.288 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:00.260+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:00.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:00.260+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:00.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:00 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:00.260+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:00.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:50:59.945400795 +0000 UTC m=+0.019602639 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:51:00.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:51:00.045335877 +0000 UTC m=+0.119537750 container create f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212) 2023-12-19T19:51:00.289 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:51:00.256302652 +0000 UTC m=+0.330504492 container init f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_CLEAN=True) 2023-12-19T19:51:00.290 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:51:00.260410541 +0000 UTC m=+0.334612372 container start f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, ceph=True, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed) 2023-12-19T19:51:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:51:00.364662707 +0000 UTC m=+0.438864550 container attach f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20231212, ceph=True, RELEASE=quincy-a88f3ed, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:51:00.789 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:00 smithi055 podman[138050]: 2023-12-19 19:51:00.551148002 +0000 UTC m=+0.625349832 container died f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-a88f3ed, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212) 2023-12-19T19:51:01.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:01 smithi055 podman[138076]: 2023-12-19 19:51:01.101589762 +0000 UTC m=+0.541383471 container remove f706643df4cc9f1c658e1a3fcea5c118d77348e7b9b7febc06e704490ce76643 (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, GIT_BRANCH=HEAD) 2023-12-19T19:51:01.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:01 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.1.service: Deactivated successfully. 2023-12-19T19:51:01.539 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:01 smithi055 systemd[1]: Stopped Ceph osd.1 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:51:01.540 INFO:journalctl@ceph.osd.1.smithi055.stdout:Dec 19 19:51:01 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.1.service: Consumed 19.612s CPU time. 2023-12-19T19:51:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.215+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:01.540 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 systemd[1]: Stopping Ceph osd.2 for 63de9ba6-9ea1-11ee-95a5-87774f69a715... 2023-12-19T19:51:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.554+0000 7fb1e3387700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-19T19:51:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.554+0000 7fb1e3387700 -1 osd.2 46 *** Got signal Terminated *** 2023-12-19T19:51:02.039 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:01 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:01.554+0000 7fb1e3387700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-19T19:51:02.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:02.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:02.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:02.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:02.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:02 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:02.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:03.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:03.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:03.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:03.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:03.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:03 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:03.205+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:04.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:04.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:04.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:04.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:04.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:04 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:04.200+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:05.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:05.238+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:05.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:05.238+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:05.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:05 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:05.238+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:06.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:06.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6804 osd.3 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:06.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:06.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6812 osd.4 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:06.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:06 smithi055 ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2[114618]: 2023-12-19T19:51:06.217+0000 7fb1e1188700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.187:6820 osd.5 ever on either front or back, first ping sent 2023-12-19T19:45:14.832923+0000 (oldest deadline 2023-12-19T19:45:34.832923+0000) 2023-12-19T19:51:07.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:07 smithi055 podman[138155]: 2023-12-19 19:51:07.031322897 +0000 UTC m=+5.587354858 container died 338fa21e9e0b5eaf529bb7d8a6dc304613f571935cb6123850293782ec059a9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-19T19:51:07.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:07 smithi055 podman[138155]: 2023-12-19 19:51:07.404336042 +0000 UTC m=+5.960367993 container cleanup 338fa21e9e0b5eaf529bb7d8a6dc304613f571935cb6123850293782ec059a9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2023-12-19T19:51:07.580 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:07 smithi055 bash[138155]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2 2023-12-19T19:51:08.025 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:07 smithi055 podman[138172]: 2023-12-19 19:51:07.664147753 +0000 UTC m=+0.625045346 container remove 338fa21e9e0b5eaf529bb7d8a6dc304613f571935cb6123850293782ec059a9a (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-a88f3ed, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, ceph=True, org.label-schema.schema-version=1.0) 2023-12-19T19:51:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:07.941480934 +0000 UTC m=+0.022362565 image pull f9ecdbaa1030a8dc3428557b1b56bd9d3af8551038d2d65c4a38bae5ee62341e quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd 2023-12-19T19:51:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:08.039973987 +0000 UTC m=+0.120855594 container create 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-a88f3ed, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:51:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:08.225969095 +0000 UTC m=+0.306850719 container init 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, ceph=True, org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-a88f3ed, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e) 2023-12-19T19:51:08.289 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:08.229202472 +0000 UTC m=+0.310084088 container start 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, CEPH_POINT_RELEASE=, RELEASE=quincy-a88f3ed, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2023-12-19T19:51:08.565 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:08.309193179 +0000 UTC m=+0.390074811 container attach 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, RELEASE=quincy-a88f3ed, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.31.3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2023-12-19T19:51:08.569 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:08 smithi055 podman[138269]: 2023-12-19 19:51:08.526114256 +0000 UTC m=+0.606995883 container died 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, ceph=True, RELEASE=quincy-a88f3ed, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2) 2023-12-19T19:51:09.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:09 smithi055 podman[138297]: 2023-12-19 19:51:09.071085797 +0000 UTC m=+0.539387601 container remove 932e16a66338a137d8dd1a3af0e8e516a6ad80b5e77f56c2553faca017f4a9fc (image=quay.ceph.io/ceph-ci/ceph@sha256:16492dd117e06e94cb8038f70cbd19bda01bc92780826dbd8a342d98670acfdd, name=ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-a88f3ed, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=4e397bbc7ff93e76025ef390087dfcea05ef676e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-19T19:51:09.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:09 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.2.service: Deactivated successfully. 2023-12-19T19:51:09.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:09 smithi055 systemd[1]: Stopped Ceph osd.2 for 63de9ba6-9ea1-11ee-95a5-87774f69a715. 2023-12-19T19:51:09.539 INFO:journalctl@ceph.osd.2.smithi055.stdout:Dec 19 19:51:09 smithi055 systemd[1]: ceph-63de9ba6-9ea1-11ee-95a5-87774f69a715@osd.2.service: Consumed 16.473s CPU time. 2023-12-19T19:51:14.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:51:14.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:51:17.873 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:51:26.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:51:26.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:51:29.937 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:51:41.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:51:41.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:51:45.009 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:51:48.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:51:48.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:51:48.081 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:51:48.082 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:51:48.082 INFO:tasks.cephadm:Teardown complete 2023-12-19T19:51:48.082 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1540, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1797, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1673, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 178, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 297, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:51:48.103 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-19T19:51:48.116 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-19T19:51:48.132 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-19T19:51:48.132 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T19:51:48.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:51:48.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:51:51.217 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:00.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:00.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:03.281 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:15.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:15.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:18.357 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:21.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:21.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:21.425 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:21.425 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:52:21.472 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:52:21.489 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-19T19:52:21.502 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-19T19:52:21.503 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T19:52:21.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:21.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:21.526 INFO:teuthology.orchestra.run.smithi055.stderr:bash: line 1: ntpq: command not found 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:=============================================================================== 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 94 +77us[ +62us] +/- 71ms 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -264us[ -279us] +/- 52ms 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 +307us[ +307us] +/- 76ms 2023-12-19T19:52:21.530 INFO:teuthology.orchestra.run.smithi055.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T19:52:24.561 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:33.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:33.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:36.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:48.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:48.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:51.697 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:54.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:54.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:54.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:52:54.770 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:52:54.770 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:52:54.789 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-19T19:52:54.809 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-19T19:52:54.810 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-19T19:52:54.823 DEBUG:teuthology.orchestra.run.smithi055:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-19T19:52:54.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:52:54.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:52:58.001 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:07.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:07.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:10.065 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:22.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:22.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:25.137 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:28.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:28.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:28.209 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:28.210 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:53:28.211 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:53:28.218 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-19T19:53:28.235 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-19T19:53:28.247 INFO:teuthology.task.internal:Duration was 3485.368873 seconds 2023-12-19T19:53:28.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-19T19:53:28.259 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-19T19:53:28.260 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T19:53:28.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:28.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:28.323 INFO:teuthology.orchestra.run.smithi055.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T19:53:31.345 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:40.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:40.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:43.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:53:55.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:53:55.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:53:58.485 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:01.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:01.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:01.553 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:01.554 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:54:01.555 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:54:01.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-19T19:54:01.596 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-19T19:54:01.596 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T19:54:01.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:01.641 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:04.721 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:13.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:13.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:16.785 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:28.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:28.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:31.857 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:34.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:34.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:34.933 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:34.934 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:54:34.934 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:54:34.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-19T19:54:34.962 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T19:54:34.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:34.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:35.002 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2023-12-19T19:54:38.033 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:54:47.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:54:47.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:54:50.097 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:02.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:02.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:05.169 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:08.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:08.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:08.241 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:08.242 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:55:08.242 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:55:08.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-19T19:55:08.271 INFO:teuthology.task.internal:Transferring archived files... 2023-12-19T19:55:08.272 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329/remote/smithi055 2023-12-19T19:55:08.273 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T19:55:08.322 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329/remote/smithi187 2023-12-19T19:55:08.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:08.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:11.377 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:20.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:20.380 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:23.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:35.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:35.443 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:38.517 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:41.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:41.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:41.585 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:41.586 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:55:41.586 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:55:41.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-19T19:55:41.620 INFO:teuthology.task.internal:Not uploading archives. 2023-12-19T19:55:41.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-19T19:55:41.630 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-19T19:55:41.631 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T19:55:41.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:41.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:41.654 INFO:teuthology.orchestra.run.smithi055.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 19 19:51 /home/ubuntu/cephtest 2023-12-19T19:55:41.654 INFO:teuthology.orchestra.run.smithi055.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 19 19:54 /home/ubuntu/cephtest/archive 2023-12-19T19:55:41.654 INFO:teuthology.orchestra.run.smithi055.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 18:55 /home/ubuntu/cephtest/archive/syslog 2023-12-19T19:55:41.655 INFO:teuthology.orchestra.run.smithi055.stdout: 265245 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 19 18:55 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T19:55:41.655 INFO:teuthology.orchestra.run.smithi055.stdout: 265244 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 19 18:55 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T19:55:41.655 INFO:teuthology.orchestra.run.smithi055.stdout: 674014 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 19:52 /home/ubuntu/cephtest/archive/audit 2023-12-19T19:55:41.655 INFO:teuthology.orchestra.run.smithi055.stdout: 674016 292 -rw------- 1 ubuntu ubuntu 297524 Dec 19 19:52 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-19T19:55:41.655 INFO:teuthology.orchestra.run.smithi055.stdout: 265280 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Dec 19 19:04 /home/ubuntu/cephtest/cephadm 2023-12-19T19:55:41.656 INFO:teuthology.orchestra.run.smithi055.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-19T19:55:44.689 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:55:53.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:55:53.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:55:56.754 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:56:08.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:56:08.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:56:11.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:56:14.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-12-19T19:56:14.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T19:56:14.901 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-12-19T19:56:14.902 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-19T19:56:14.903 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi187 2023-12-19T19:56:14.920 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-19T19:56:14.934 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-19T19:56:14.946 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2023-12-19T19:56:14.947 INFO:teuthology.nuke:Checking targets against current locks 2023-12-19T19:56:14.977 DEBUG:teuthology.nuke:shortname: smithi055 2023-12-19T19:56:14.977 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T19:56:15.003 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:46:55.890668', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNIshYVrt2DKHlTX/Sdu2dvCpDQ2/Iy9qOzMXB3cvi+FK4M2A6iNfCYWwz6OgsT3BhpLadM6UfQYwSI4nxbTwNw='} 2023-12-19T19:56:15.006 DEBUG:teuthology.nuke:shortname: smithi187 2023-12-19T19:56:15.006 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T19:56:15.032 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-19_15:49:15-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7497329', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 18:46:55.892333', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9YjkQCn2MGGsyTxyOuNT5q8xy0ZK7SdXyzUPvjnz69s0cohuF7xFzefH1fvVGlT8uDbMAex4e1Xs6nPfrfcZE='} 2023-12-19T19:56:15.054 INFO:teuthology.orchestra.console.smithi055:Power off 2023-12-19T19:56:15.054 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T19:56:15.079 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2023-12-19T19:56:15.080 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:15.096 INFO:teuthology.orchestra.console.smithi187:Power off 2023-12-19T19:56:15.096 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-19T19:56:15.106 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-19T19:56:15.119 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2023-12-19T19:56:15.120 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:15.143 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-19T19:56:19.107 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:19.144 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:19.217 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-19T19:56:19.254 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-19T19:56:23.222 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:23.255 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:23.333 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-19T19:56:23.365 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-12-19T19:56:27.337 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:27.366 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-19T19:56:27.448 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2023-12-19T19:56:27.476 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-12-19T19:56:27.549 INFO:teuthology.orchestra.console.smithi055:Power off completed 2023-12-19T19:56:27.577 INFO:teuthology.orchestra.console.smithi187:Power off completed 2023-12-19T19:56:27.683 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3485.3688728809357 failure_reason: 'Command failed on smithi055 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 63de9ba6-9ea1-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dbeaef3cb93e4c5f80a219e12f39d428 status: fail success: false 2023-12-19T19:56:27.684 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T19:56:27.767 INFO:teuthology.run:FAIL